ODK Collect

ODK Collect is an Android app for filling out forms. It is designed to be used in resource-constrained environments with challenges such as unreliable connectivity or power infrastructure. ODK Collect is part the ODK project, a free and open-source set of tools which help organizations author, field, and manage mobile data collection solutions. Learn more about ODK and its history here and read about example ODK deployments here.

ODK Collect renders forms that are compliant with the ODK XForms standard, a subset of the XForms 1.1 standard with some extensions. The form parsing is done by the JavaRosa library which Collect includes as a dependency.

Please note that the master branch reflects ongoing development and is not production-ready.

Table of Contents

• Learn more about ODK Collect
• Release cycle
• Downloading builds
• Suggesting new features
• Contributing
  • Contributing code
  • Contributing translations
  • Contributing testing
• Developing
  • Setting up your development environment
  • Testing a form without a server
  • Using APIs for local development
  • Debugging JavaRosa
  • Troubleshooting
• Creating signed releases for Google Play Store

Learn more about ODK Collect

• ODK website: https://getodk.org
• ODK Collect usage documentation: https://docs.getodk.org/collect-intro/
• ODK forum: https://forum.getodk.org
• ODK developer Slack chat: https://slack.getodk.org

Release cycle

Releases can be requested by any community member and generally happen every 2 months.

Before release we perform a "code freeze" (we stop merging pull requests). A new beta will then be released to the Play Store by @lognaturel and that will be used for regression testing by @getodk/testers. If any problems are found, the release is blocked until we can merge fixes. Regression testing should continue on the original beta build (rather than a new one with fixes) unless problems block the rest of testing. Once the process is complete, @lognaturel pushes the releases to the Play Store following these instructions. The code is "unfrozen" after a short grace period to make hot fixing easier.

At the beginning of each release cycle, @grzesiek2010 updates all dependencies that have compatible upgrades available and ensures that the build targets the latest SDK.

Downloading builds

Per-commit debug builds can be found on CircleCI. Login with your GitHub account, click the build you'd like, then find the APK in the Artifacts tab.

If you are looking to use ODK Collect, we strongly recommend using the Play Store build. Current and previous production builds can be found in Releases.

Suggesting new features

We try to make sure that all issues in the issue tracker are as close to fully specified as possible so that they can be closed by a pull request. Feature suggestions should be described in the forum Features category and discussed by the broader user community. Once there is a clear way forward, issues should be filed on the relevant repositories. More controversial features will be discussed as part of the Technical Steering Committee's roadmapping process.

Contributing code

Any and all contributions to the project are welcome. ODK Collect is used across the world primarily by organizations with a social purpose so you can have real impact!

Issues tagged as good first issue should be a good place to start. There are also currently many issues tagged as needs reproduction which need someone to try to reproduce them with the current version of ODK Collect and comment on the issue with their findings.

If you're ready to contribute code, see the contribution guide.

Contributing translations

If you know a language other than English, consider contributing translations through Transifex.

Translations are updated right before the first beta for a release and before the release itself. To update translations, download the zip from https://www.transifex.com/getodk/collect/strings/. The contents of each folder then need to be moved to the Android project folders. A quick script like the one in this gist can help. We currently copy everything from Transifex to minimize manual intervention. Sometimes translation files will only get comment changes. When new languages are updated in Transifex, they need to be added to the script above. Additionally, ApplicationConstants.TRANSLATIONS_AVAILABLE needs to be updated. This array provides the choices for the language preference in settings. Ideally the list could be dynamically generated.

Contributing testing

All pull requests are verified on the following devices (ordered by Android version):

• Huawei Y560-L01 - Android 5.1.1
• Sony Xperia Z3 D6603 - Android 6.0.1 (used irregularly)
• Samsung Galaxy S7 SM-G930F - Android 7.0.0 (used irregularly)
• Motorola Moto G4 Play - Android 7.1.1 (used irregularly)
• LG Nexus 5X - Android 8.1
• Xiaomi Redmi 7 - Android 9.0 (used irregularly)
• Samsung Galaxy M11 SM-M115F/DSN - Android 10.0
• Google Pixel 3a - Android 11.0

Our regular code contributors use these devices (ordered by Android version):

• Samsung Galaxy Tab SM-T285 - Android 5.1.1 @lognaturel
• Motorola G 5th Gen XT1671 - Android 7.0 @lognaturel

The best way to help us test is to build from source! If you aren't a developer and want to help us test release candidates, join the beta program!

Testing checklists can be found on the Collect testing plan.

If you have finished testing a pull request, please use a template from Testing result templates to report your insights.

Setting up your development environment

1. Download and install Git and add it to your PATH

2. Download and install Android Studio

3. Fork the collect project (why and how to fork)

4. Clone your fork of the project locally. At the command line:

    git clone https://github.com/YOUR-GITHUB-USERNAME/collect
   

   If you prefer not to use the command line, you can use Android Studio to create a new project from version control using https://github.com/YOUR-GITHUB-USERNAME/collect.

5. Use Android Studio to import the project from its Gradle settings. To run the project, click on the green arrow at the top of the screen.

6. Windows developers: continue configuring Android Studio with the steps in this document: Developing ODK Collect on Windows.

7. Make sure you can run unit tests by running everything under collect_app/src/test/java in Android Studio or on the command line:

   ./gradlew testDebug
   

8. Make sure you can run instrumented tests by running everything under collect_app/src/androidTest/java in Android Studio or on the command line:

   ./gradlew connectedAndroidTest
   

   Note: You can see the emulator setup used on CI in .circleci/config.yml.

Customizing the development environment

Changing JVM heap size

You can customize the heap size that is used for compiling and running tests. Increasing these will most likely speed up compilation and tests on your local machine. The default values are specified in the project's gradle.properties and this can be overriden by your user level gradle.properties (found in your GRADLE_USER_HOME directory). An example gradle.properties that would give you a heap size of 4GB (rather than the default 1GB) would look like:

org.gradle.jvmargs=-Xmx4096m

Testing a form without a server

When you first run Collect, it is set to download forms from https://demo.getodk.org/, the demo server. You can sometimes verify your changes with those forms but it can also be helpful to put a specific test form on your device. Here are some options for that:

1. The All Widgets form from the default server is here. You can also try example forms and test forms or make your own.

2. Convert the XLSForm (xlsx) to XForm (xml). Use the ODK website or XLSForm Offline or pyxform.

3. Once you have the XForm, use adb to push the form to your device (after enabling USB debugging) or emulator.

   adb push my_form.xml /sdcard/Android/data/org.odk.collect.android/files/forms
   

4. Launch ODK Collect and tap Fill Blank Form. The new form will be there.

Using APIs for local development

Certain functions in ODK Collect depend on cloud services that require API keys or authorization steps to work. Here are the steps you need to take in order to use these functions in your development builds.

Google Drive and Sheets APIs: When the "Google Drive, Google Sheets" option is selected in the "Server" settings, ODK Collect uses these APIs to store submitted form data in Google Sheets and submitted media in Google Drive. To enable these APIs:

1. Create and configure a Google API project for Google Sign-in using the "Configure Project" button found here.
   1. Choose whatever you'd like for the project and product name
   2. Select "Android" for "What are you calling from?"
   3. Enter "org.odk.collect.android" for package name
   4. Enter your debug key's SHA1 certificate fingerprint as the SHA1 (more info on that here)
   5. Copy the displayed "Client ID" into client_id (under oauth_client) in google-services.json
2. Enable the Google Drive API.
3. Enable the Google Sheets API.

Google Maps API: When the "Google Maps SDK" option is selected in the "User interface" settings, ODK Collect uses the Google Maps API for displaying maps in the geospatial widgets (GeoPoint, GeoTrace, and GeoShape). To enable this API:

1. Get a Google Maps API key. Note that this requires a credit card number, though the card will not be charged immediately; some free API usage is permitted. You should carefully read the terms before providing a credit card number.
2. Edit or create secrets.properties and set the GOOGLE_MAPS_API_KEY property to your API key. You should end up with a line that looks like this: GOOGLE_MAPS_API_KEY=AIbzvW8e0ub...

Mapbox Maps SDK for Android: When the "Mapbox SDK" option is selected in the "User interface" settings, ODK Collect uses the Mapbox SDK for displaying maps in the geospatial widgets (GeoPoint, GeoTrace, and GeoShape). To enable this API:

1. Create a Mapbox account. Note that signing up with the "Pay-As-You-Go" plan does not require a credit card. Mapbox provides free API usage up to the monthly thresholds documented at https://www.mapbox.com/pricing. If your usage exceeds these thresholds, you will receive e-mail with instructions on how to add a credit card for payment; services will remain live until the end of the 30-day billing term, after which the account will be deactivated and will require a credit card to reactivate.
2. Find your access token on your account page - it should be in "Tokens" as "Default public token".
3. Edit or create secrets.properties and set the MAPBOX_ACCESS_TOKEN property to your access token. You should end up with a line that looks like this: MAPBOX_ACCESS_TOKEN=pk.eyJk3bumVp4i...
4. Create a new secret token with the "DOWNLOADS:READ" secret scope and then add it to secrets.properties as MAPBOX_DOWNLOADS_TOKEN.

Note: Mapbox will not be available as an option in compiled versions of Collect unless you follow the steps above. Mapbox will also not be available on x86 devices as the native libraries are excluded to reduce the APK size. If you need to use an x86 device, you can force the build to include x86 libs by include the x86Libs Gradle parameter. For example, to build a debug APK with x86 libs: ./gradlew assembleDebug -Px86Libs.

Debugging JavaRosa

JavaRosa is the form engine that powers Collect. If you want to debug or change that code while running Collect you can deploy it locally with Maven (you'll need mvn and sed installed):

1. Build and install your changes of JavaRosa (into your local Maven repo):

./gradlew installLocal

1. Change implementation(Dependencies.javarosa) in Collect's build.gradle to implementation(Dependencies.javarosa_local)

Troubleshooting

Error when running Robolectric tests from Android Studio on macOS: build/intermediates/bundles/debug/AndroidManifest.xml (No such file or directory)

Configure the default JUnit test runner configuration in order to work around a bug where IntelliJ / Android Studio does not set the working directory to the module being tested. This can be accomplished by editing the run configurations, Defaults -> JUnit and changing the working directory value to $MODULE_DIR$.

Source: Robolectric Wiki.

Android Studio Error: SDK location not found. Define location with sdk.dir in the local.properties file or with an ANDROID_HOME environment variable.

When cloning the project from Android Studio, click "No" when prompted to open the build.gradle file and then open project.

Execution failed for task ':collect_app:transformClassesWithInstantRunForDebug'.

We have seen this problem happen in both IntelliJ IDEA and Android Studio, and believe it to be due to a bug in the IDE, which we can't fix. As a workaround, turning off Instant Run will usually avoid this problem. The problem is fixed in Android Studio 3.5 with the new Apply Changes feature.

Moving to the main view if user minimizes the app

If you build the app on your own using Android Studio (Build -> Build APK) and then install it (from an .apk file), you might notice this strange behaviour thoroughly described: #1280 and #1142.

This problem occurs building other apps as well.

gradlew Failure: FAILURE: Build failed with an exception.

If you encounter an error similar to this when running gradlew:

FAILURE: Build failed with an exception

What went wrong:
A problem occurred configuring project ':collect_app'.
> Failed to notify project evaluation listener.
   > Could not initialize class com.android.sdklib.repository.AndroidSdkHandler

You may have a mismatch between the embedded Android SDK Java and the JDK installed on your machine. You may wish to set your JAVA_HOME environment variable to that SDK. For example, on macOS:

export JAVA_HOME="/Applications/Android\ Studio.app/Contents/jre/Contents/Home/"

Note that this change might cause problems with other Java-based applications (e.g., if you uninstall Android Studio).

gradlew Failure: java.lang.NullPointerException (no error message).

If you encounter the java.lang.NullPointerException (no error message). when running gradlew, please make sure your Java version for this project is Java 8.

This can be configured under File > Project Structure in Android Studio, or by editing $USER_HOME/.gradle/gradle.properties to set org.gradle.java.home=(path to JDK home) for command-line use.

Unable to resolve artifact: Missing while running tests

This is encountered when Robolectric has problems downloading the jars it needs for different Android SDK levels. If you keep running into this you can download the JARs locally and point Robolectric to them by doing:

./download-robolectric-deps.sh

Creating signed releases for Google Play Store

Maintainers keep a folder with a clean checkout of the code and use jenv.be in that folder to ensure compilation with Java 11.

Release prerequisites:

• alocal.properties file in the root folder with the following:

  sdk.dir=/path/to/android/sdk
  

• the keystore file and passwords

• a secrets.properties file in the root project folder folder with the following:

  // secrets.properties
  RELEASE_STORE_FILE=/path/to/collect.keystore
  RELEASE_STORE_PASSWORD=secure-store-password
  RELEASE_KEY_ALIAS=key-alias
  RELEASE_KEY_PASSWORD=secure-alias-password
  

• a google-services.json file in the collect_app/src/odkCollectRelease folder. The contents of the file are similar to the contents of collect_app/src/google-services.json.

Release checklist:

• update translations
• tag the build by adding a release. Tags for full releases must have the format vX.X.X. Tags for beta releases must have the format vX.X.X-beta.X.
• run ./gradlew assembleOdkCollectRelease. If successful, a signed release will be at collect_app/build/outputs/apk.
• verify the apk size. If it has grown more than a few hundred kilobytes, discuss with the dev team.
• verify a basic "happy path": scan a QR code to configure a new project, get a blank form, fill it, open the form map (confirms that the Google Maps key is correct), send form
• verify new APK can be installed as update to previous version and that above "happy path" works in that case also
• create and publish scheduled forum post with release description
• write Play Store release notes, include link to forum post
• upload to Play Store
• if there was an active beta before release (this can happen with point releases), publish a new beta release to replace the previous one which was disabled by the production release
• attach APK to previously created Github Release with the name ODK-Collect-vX.X.X.apk
• backup dependencies for the release by downloading the vX.X.X.tar artifact from the create_dependency_backup job on Circle CI (for the release commit) and then uploading it to the "Collect Dependency Backups" folder in GetODK's Google Drive

Compiling a previous release using backed-up dependencies

1. Download the .tar for relevant release tag
2. Extract .local-m2 into the project directory:

   tar -xf maven.tar -C <collect project directory>

The project will now be able to fetch dependencies that are no longer available (but were used to compile the release) from the .local-m2 Maven repo.