Implemented CameraX and Data Binding in Object Detection App #341

sayannath · 2021-08-07T09:54:05Z

Hello!

I have replaced the Camera and Camera2 API implementations with fragments with CameraX.
Implemented data binding as well.
Refactored the code according to that
Managed Image to Bitmap Conversion for tf_support
Modified test code for the app

lintian06 · 2021-08-10T03:53:36Z

...on/android/app/src/androidTest/java/org/tensorflow/lite/examples/detection/DetectorTest.java

@@ -14,151 +14,150 @@
 * limitations under the License.
 */

-package org.tensorflow.lite.examples.detection;
-


Please don't delete the test, but make it work. We need to test the change and make sure it works as intended.

Check this example if it can help you:
https://android.googlesource.com/platform/frameworks/support/+/refs/heads/androidx-arch-core-release/camera/camera2/src/androidTest/java/androidx/camera/camera2/UseCaseCombinationTest.java

Sure, will add this asap!

lintian06 · 2021-08-10T04:02:47Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

@@ -17,278 +17,307 @@
 package org.tensorflow.lite.examples.detection;

 import android.Manifest;


Imports should be sorted in alphabetic order.

lintian06 · 2021-08-10T04:04:12Z

...c/main/java/org/tensorflow/lite/examples/detection/tflite/TFLiteObjectDetectionAPIModel.java

+          final String labelFilename,
+          final int inputSize,
+          final boolean isQuantized)
+          throws IOException {
    return new TFLiteObjectDetectionAPIModel(context, modelFilename);


Please format following 2 spaces indentation.

lintian06 · 2021-08-10T04:04:30Z

...c/main/java/org/tensorflow/lite/examples/detection/tflite/TFLiteObjectDetectionAPIModel.java

+    int cropSize = min(width, height);
+
+    ImageProcessingOptions imageOptions =
+            ImageProcessingOptions.builder()


Ditto (indent).

lintian06 · 2021-08-10T04:04:45Z

...c/main/java/org/tensorflow/lite/examples/detection/tflite/TFLiteObjectDetectionAPIModel.java

+                      "" + cnt++,
+                      detection.getCategories().get(0).getLabel(),
+                      detection.getCategories().get(0).getScore(),
+                      detection.getBoundingBox()));


Ditto (indent).

lintian06 · 2021-08-10T04:04:55Z

...c/main/java/org/tensorflow/lite/examples/detection/tflite/TFLiteObjectDetectionAPIModel.java

+          final int dstWidth,
+          final int dstHeight,
+          final int applyRotation,
+          final boolean maintainAspectRatio) {


Ditto (indent).

lintian06 · 2021-08-10T04:05:31Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-      LOGGER.e(e, "Exception!");
-      return;
+      detector =
+              TFLiteObjectDetectionAPIModel.create(


No line break please.

lintian06 · 2021-08-10T04:05:54Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

+      e.printStackTrace();
+      LOGGER.e(e, "Exception initializing Detector!");
+      Toast toast =
+              Toast.makeText(


Ditto (no line break)

lintian06 · 2021-08-10T04:06:20Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

+                switch (newState) {
+                  case BottomSheetBehavior.STATE_HIDDEN:
+                    break;
+                  case BottomSheetBehavior.STATE_EXPANDED: {


Please follow 2 spaces indentation.

Thank you for the PR!

Since it contains UI change, can you show a screenshot of current running app? Thank you!

Yes sure I will attach a Screenshot in the upcoming Commit

lintian06 · 2021-08-10T04:06:34Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-            sheetBehavior.setPeekHeight(height);
-          }
-        });
+            new ViewTreeObserver.OnGlobalLayoutListener() {


Ditto (indent).

lintian06 · 2021-08-10T04:07:57Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java


    if (hasPermission()) {
-      setFragment();
+      //Start CameraX


Comment should start with a space, like // Start CameraX. Please do the same for the other occurrences.

lintian06 · 2021-08-10T04:09:51Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-    return yuvBytes[0];
-  }
+  private void onStartCameraX(Size size, final int rotation) {
+    final float textSizePx =


What does px in textSizePx mean? You can call it textSize.

lintian06 · 2021-08-10T04:11:50Z

...ction/android/app/src/main/java/org/tensorflow/lite/examples/detection/env/BorderedText.java

@@ -79,7 +79,7 @@ public void drawText(final Canvas canvas, final float posX, final float posY, fi
  }

  public void drawText(
-      final Canvas canvas, final float posX, final float posY, final String text, Paint bgPaint) {
+          final Canvas canvas, final float posX, final float posY, final String text, Paint bgPaint) {


Please recover the original indent. (spaces = 2)

lintian06 · 2021-08-10T04:11:56Z

...ction/android/app/src/main/java/org/tensorflow/lite/examples/detection/env/BorderedText.java

@@ -117,12 +117,12 @@ public void setAlpha(final int alpha) {
  }



Please recover the original indent. (spaces = 2)

lintian06 · 2021-08-10T04:12:24Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-        return 0;
-    }
+  protected void setUseNNAPI(final boolean isChecked) {
+    runInBackground(


Please format this. (spaces = 2)

lintian06 · 2021-08-10T04:12:39Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

@@ -341,13 +370,15 @@ protected synchronized void runInBackground(final Runnable r) {
    }
  }

+
  @Override


Please recover the original indent. (spaces = 2)

lintian06 · 2021-08-10T04:13:03Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-          public void run() {
-            ImageUtils.convertYUV420SPToARGB8888(bytes, previewWidth, previewHeight, rgbBytes);
+    binding.trackingOverlay.addCallback(
+            canvas -> {


Please format this with the right indentation. (spaces = 2)

lintian06 · 2021-08-10T04:18:26Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-            camera.addCallbackBuffer(bytes);
-            isProcessingFrame = false;
+
+          if (!isProcessingFrame) {


Extra 2 spaces. The block doesn't align with the previous one.

lintian06 · 2021-08-10T04:19:41Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-        image.close();
-        return;
+      } catch (ExecutionException | InterruptedException e) {
+        e.printStackTrace();


Please log the error, instead of printing it.

lintian06 · 2021-08-10T04:20:29Z

...droid/app/src/main/java/org/tensorflow/lite/examples/detection/tracking/MultiBoxTracker.java

-    Color.parseColor("#AA33AA"),
-    Color.parseColor("#0D0068")
+          Color.BLUE,
+          Color.RED,


Please recover the original indentation. (spaces = 2)

lintian06 · 2021-08-10T04:21:21Z

...droid/app/src/main/java/org/tensorflow/lite/examples/detection/tracking/MultiBoxTracker.java

@@ -144,13 +145,11 @@ public synchronized void draw(final Canvas canvas) {
      canvas.drawRoundRect(trackedPos, cornerSize, cornerSize, boxPaint);

      final String labelString =
-          !TextUtils.isEmpty(recognition.title)
-              ? String.format("%s %.2f", recognition.title, (100 * recognition.detectionConfidence))


Please recover the original indentation. (spaces = 2)

lintian06 · 2021-08-10T04:21:31Z

...droid/app/src/main/java/org/tensorflow/lite/examples/detection/tracking/MultiBoxTracker.java

      borderedText.drawText(
-          canvas, trackedPos.left + cornerSize, trackedPos.top, labelString + "%", boxPaint);


Please recover the original indentation. (spaces = 2)

lintian06 · 2021-08-10T04:21:52Z

...droid/app/src/main/java/org/tensorflow/lite/examples/detection/tracking/MultiBoxTracker.java

@@ -170,7 +169,7 @@ private void processResults(final List<Recognition> results) {
      rgbFrameToScreen.mapRect(detectionScreenRect, detectionFrameRect);

      logger.v(
-          "Result! Frame: " + result.getLocation() + " mapped to screen:" + detectionScreenRect);
+              "Result! Frame: " + result.getLocation() + " mapped to screen:" + detectionScreenRect);


Please recover the original indentation. (spaces = 2)

lintian06

Thank you for the PR!

Since it contains UI change, can you show a screenshot of current running app? Thank you!

khanhlvg · 2021-08-10T03:16:03Z

...on/android/app/src/androidTest/java/org/tensorflow/lite/examples/detection/DetectorTest.java

-    return result;
-  }
-}
+//package org.tensorflow.lite.examples.detection;


Please make the test passed rather than commenting it out.

@sayannath Check this example if it can help you:
https://android.googlesource.com/platform/frameworks/support/+/refs/heads/androidx-arch-core-release/camera/camera2/src/androidTest/java/androidx/camera/camera2/UseCaseCombinationTest.java

khanhlvg · 2021-08-10T03:20:21Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

@@ -378,173 +409,59 @@ private void requestPermission() {
                CameraActivity.this,
                "Camera permission is required for this demo",
                Toast.LENGTH_LONG)
-            .show();
+                .show();


Please revert all changes that only change the indent like this. It make the PR very hard to review because there're a lot of changes that don't have any effect.

khanhlvg · 2021-08-10T03:22:51Z

lite/examples/object_detection/android/app/build.gradle

+    androidTestImplementation 'androidx.test:rules:1.4.0'
+
+    // CameraX dependencies
+    def camerax_version = "1.1.0-alpha05"


Please use latest stable version

khanhlvg · 2021-08-10T03:24:17Z

lite/examples/object_detection/android/app/build.gradle

+    // CameraX Lifecycle Library
+    implementation "androidx.camera:camera-lifecycle:$camerax_version"
+    // CameraX View class
+    implementation "androidx.camera:camera-view:1.0.0-alpha25"


Do you really need androidx.camera:camera-view? I don't see you use it anywhere in the repo.

Using the androidx.camera:camera-view
To Connect the preview use case to the previewView

preview.setSurfaceProvider(binding.previewView.getSurfaceProvider());

khanhlvg · 2021-08-10T03:29:11Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

+                        lastProcessingTimeMs = SystemClock.uptimeMillis() - startTime;
+                        LOGGER.e("Degrees: %s", results);
+
+                        float minimumConfidence = MINIMUM_CONFIDENCE_TF_OD_API;


This chunk of code doesn't do anything.

float minimumConfidence = MINIMUM_CONFIDENCE_TF_OD_API; if (MODE == DetectorMode.TF_OD_API) { minimumConfidence = MINIMUM_CONFIDENCE_TF_OD_API; }

khanhlvg · 2021-08-10T03:30:55Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

  private BottomSheetBehavior<LinearLayout> sheetBehavior;

-  protected TextView frameValueTextView, cropValueTextView, inferenceTimeTextView;
+  private enum DetectorMode {
+    TF_OD_API;


What is this enum for? If it isn't used, remove it.

It was previously there in the code so I felt like keeping it.

Please delete as it doesn't do anything.

khanhlvg · 2021-08-10T03:31:41Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

+
+  private static final int TF_OD_API_INPUT_SIZE = 300;
+  private static final boolean TF_OD_API_IS_QUANTIZED = true;
+  private static final String TF_OD_API_MODEL_FILE = "detect.tflite";


Why do you prefix variable names with TF_OD_API? If there's no special reason for that, please remove.

I have taken the reference from here, from the base repository I mean.

Let's delete the prefix then because it's confusing.

khanhlvg · 2021-08-10T03:32:25Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

+                int height = binding.bottomSheetLayout.gestureLayout.getMeasuredHeight();
+                sheetBehavior.setPeekHeight(height);
+              }
+            });
    sheetBehavior.setHideable(false);

    sheetBehavior.setBottomSheetCallback(


This method was deprecated. Please update.

khanhlvg · 2021-08-10T04:29:59Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

  private static final Logger LOGGER = new Logger();

  private static final int PERMISSIONS_REQUEST = 1;

  private static final String PERMISSION_CAMERA = Manifest.permission.CAMERA;
  protected int previewWidth = 0;
  protected int previewHeight = 0;
+
  private boolean debug = false;


Add a comment to explain what this is for.

khanhlvg · 2021-08-11T23:20:36Z

interpreter build variants doesn't build. Please fix the build error and all warnings.

khanhlvg · 2021-08-11T23:25:37Z

...tection/android/app/src/main/java/org/tensorflow/lite/examples/detection/CameraActivity.java

-  protected byte[] getLuminance() {
-    return yuvBytes[0];
-  }
+  private void onStartCameraX(Size size, final int rotation) {


size is always DESIRED_ANALYSIS_SIZE so you can use the value directly inside the method rather than passing in a parameter.

sayannath · 2021-08-12T05:25:19Z

Hi @lintian06 and @khanhlvg
I have attached the screenshot of the app.

sayannath · 2021-08-13T19:09:50Z

Hi @khanhlvg and @lintian06 I have added the test. @farmaker47 helped me.

Implemented CameraX and Data Binding

a008ebd

sayannath requested review from khanhlvg, lintian06 and yyoon as code owners August 7, 2021 09:54

google-ml-butler bot added size:XL CL Change Size: Extra Large awaiting review labels Aug 7, 2021

google-cla bot added the cla: yes CLA has been signed label Aug 7, 2021

Suggestions Implemented

8bd41b9

sayannath changed the title ~~Implemented CameraX and Data Binding~~ Implemented CameraX and Data Binding in Object Detection App Aug 8, 2021

lintian06 reviewed Aug 10, 2021

View reviewed changes

khanhlvg requested changes Aug 10, 2021

View reviewed changes

sayannath and others added 4 commits August 11, 2021 18:26

Incorporated the feedbacks, TODO: Writing the test

494cd5e

Merge branch 'tensorflow:master' into master

30fa3b8

Depricated Code Improved! TODO: To write the test

ccdbd90

Merge branch 'master' of https://github.com/sayannath/examples

7c18d5e

khanhlvg reviewed Aug 11, 2021

View reviewed changes

Feedback Round II TODO: Working on Test

1828a33

sayannath added 2 commits August 13, 2021 18:20

TODO: Refactoring the Test

fb09e0e

Adding Test for Object Detection

d37ea72

sayannath requested review from khanhlvg and lintian06 August 13, 2021 19:10

Merge branch 'tensorflow:master' into master

26e77b3

		@@ -17,278 +17,307 @@
		package org.tensorflow.lite.examples.detection;

		import android.Manifest;

		@@ -117,12 +117,12 @@ public void setAlpha(final int alpha) {
		}

		borderedText.drawText(
		canvas, trackedPos.left + cornerSize, trackedPos.top, labelString + "%", boxPaint);

Implemented CameraX and Data Binding in Object Detection App #341

Are you sure you want to change the base?

Implemented CameraX and Data Binding in Object Detection App #341

Conversation

sayannath commented Aug 7, 2021 • edited Loading

lintian06 Aug 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lintian06 Aug 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lintian06 Aug 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lintian06 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayannath Aug 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khanhlvg commented Aug 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayannath commented Aug 12, 2021 • edited Loading

sayannath commented Aug 13, 2021 • edited Loading

sayannath commented Aug 7, 2021 •

edited

Loading

lintian06 Aug 10, 2021 •

edited

Loading

lintian06 Aug 10, 2021 •

edited

Loading

lintian06 Aug 10, 2021 •

edited

Loading

sayannath Aug 10, 2021 •

edited

Loading

khanhlvg commented Aug 11, 2021 •

edited

Loading

sayannath commented Aug 12, 2021 •

edited

Loading

sayannath commented Aug 13, 2021 •

edited

Loading