Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug report] Android Platform query poor performance #1255

Open
6a209 opened this issue Jan 10, 2025 · 5 comments
Open

[Bug report] Android Platform query poor performance #1255

6a209 opened this issue Jan 10, 2025 · 5 comments

Comments

@6a209
Copy link

6a209 commented Jan 10, 2025

Version

3.2.0

Platforms

Android

Device Model

Android 15

flutter info

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.24.5, on macOS 14.6.1 23G93 darwin-arm64)
[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/to/macos-android-setup for more details.
[!] Xcode - develop for iOS and macOS (Xcode 16.1)
    ✗ Unable to get list of installed Simulator runtimes.
    ! CocoaPods 1.11.2 out of date (1.13.0 is recommended).
        CocoaPods is a package manager for iOS or macOS platform code.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/to/platform-plugins
      To update CocoaPods, see https://guides.cocoapods.org/using/getting-started.html#updating-cocoapods
[✓] Chrome - develop for the web
[✓] Android Studio (version 2024.2)
[✓] IntelliJ IDEA Community Edition (version 2022.1.3)
[✓] VS Code (version 1.95.3)
[✓] Connected device (4 available)
[✓] Network resources

! Doctor found issues in 2 categories.

How to reproduce?

I use filterOption is :

final filterOptionGroup = FilterOptionGroup(
imageOption: const FilterOption(needTitle: true),
videoOption: const FilterOption(needTitle: true),
updateTimeCond: updateTimeCond);

Logs

This is my query log : 

bucket_id IS NOT NULL  AND ( ( ( media_type = 1  AND width >= 0 AND width <= 100000 AND height >= 0 AND height 
<=100000 )OR ( media_type = 3 AND duration >=0 AND duration <=86400000 ) ) AND ( date_added >= 0 AND 
date_added <= 1736476689 ) AND ( date_modified >= -28800 AND date_modified <= 1736476689 ) AND (( media_type 
= 1 AND width > 0 AND height > 0 ) OR ( media_type = 3 )) ) 


This seems to have some issues, such as

"( media_type = 1  AND width >= 0 AND width <= 100000 AND height >= 0 AND height <=100000) "

"( media_type = 1 AND width > 0 AND height > 0 )"

It looks like these two conditions are repeated

Example code (optional)

No response

Contact

No response

@fluttercandies-dev
Copy link

AI Summary: The user reports a bug in version 3.2.0 on Android where two SQL conditions appear to be duplicated in the query log.

@AlexV525
Copy link
Member

Please provide a complete minimal reproducible example.

@6a209 6a209 changed the title [Bug report] [Bug report] Android Platform query poor performance Jan 14, 2025
@6a209
Copy link
Author

6a209 commented Jan 14, 2025

@AlexV525

the sample code like this

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:photo_manager/photo_manager.dart';

class PageTest extends StatefulWidget {
  @override
  State<PageTest> createState() => _PageTestState();
}

class _PageTestState extends State<PageTest> {
  @override
  void initState() {
    super.initState();
    PhotoManager.setLog(true);
    backupTest();
  }

  @override
  Widget build(BuildContext context) {
    return Container(
      color: Colors.blueAccent,
      child: const Center(
        child: Text('BackupTest'),
      ),
    );
  }

  Future<List<AssetEntity>> backupTest() async {
    final sw = Stopwatch()..start();
    final List<AssetEntity> backupAssets = [];
    final filterOptionGroup = FilterOptionGroup(
      imageOption: const FilterOption(needTitle: true),
      videoOption: const FilterOption(needTitle: true,),
    );
    final backupAlbums = await PhotoManager.getAssetPathList(
        onlyAll: true, filterOption: filterOptionGroup);

    for (final backupAlbum in backupAlbums) {
      int page = 0;
      final pageSize = 1000;
      bool isEnd = false;

      while (!isEnd) {
        final queryStopWatch = Stopwatch()..start();
        final assetList =
            await backupAlbum.getAssetListPaged(page: page, size: pageSize);
        queryStopWatch.stop();
        print('query time ${queryStopWatch.elapsedMilliseconds}');
        backupAssets.addAll(assetList);
        page += 1;
        if (assetList.length < pageSize) {
          isEnd = true;
        }
      }
    }
    sw.stop();
    print('total time ${sw.elapsedMilliseconds}');
    return backupAssets;
  }
}


1. the query is too complex

the log is

I/PhotoManager(16493): uri: content://media/external/file
I/PhotoManager(16493): projection: _display_name, _data, _id, title, bucket_id, bucket_display_name, width, height, orientation, date_added, date_modified, mime_type, datetaken, duration, media_type, relative_path
I/PhotoManager(16493): selection: bucket_id IS NOT NULL  AND ( ( ( media_type = ?  AND width >= ? AND width <= ? AND height >= ? AND height <=? )OR ( media_type = ? AND duration >=? AND duration <=? ) ) AND ( date_added >= ? AND date_added <= ? )  AND (( media_type = 1 AND width > 0 AND height > 0 ) OR ( media_type = 3 )) )
I/PhotoManager(16493): selectionArgs: 1, 0, 100000, 0, 100000, 3, 0, 86400000, 0, 1736834584
I/PhotoManager(16493): sortOrder: _id desc
I/PhotoManager(16493): sql: bucket_id IS NOT NULL  AND ( ( ( media_type = 1  AND width >= 0 AND width <= 100000 AND height >= 0 AND height <=100000 )OR ( media_type = 3 AND duration >=0 AND duration <=86400000 ) ) AND ( date_added >= 0 AND date_added <= 1736834584 )  AND (( media_type = 1 AND width > 0 AND height > 0 ) OR ( media_type = 3 )) )
I/PhotoManager(16493): cursor count: **57639**
I/flutter (16493): query time 481

This has taken a lot of time, so I change the filter code to

    final filterOptionGroup = FilterOptionGroup(
      imageOption: const FilterOption(
        needTitle: true,
        sizeConstraint: SizeConstraint(ignoreSize: true),
      ),
      videoOption: const FilterOption(
          needTitle: true, sizeConstraint: SizeConstraint(ignoreSize: true)),
    );

the log:

I/PhotoManager(18617): uri: content://media/external/file
I/PhotoManager(18617): projection: _display_name, _data, _id, title, bucket_id, bucket_display_name, width, height, orientation, date_added, date_modified, mime_type, datetaken, duration, media_type, relative_path
I/PhotoManager(18617): selection: bucket_id IS NOT NULL  AND ( ( ( media_type = ?  )OR ( media_type = ? AND duration >=? AND duration <=? ) ) AND ( date_added >= ? AND date_added <= ? )   )
I/PhotoManager(18617): selectionArgs: 1, 3, 0, 86400000, 0, 1736835350
I/PhotoManager(18617): sortOrder: _id desc
I/PhotoManager(18617): sql: bucket_id IS NOT NULL  AND ( ( ( media_type = 1  )OR ( media_type = 3 AND duration >=0 AND duration <=86400000 ) ) AND ( date_added >= 0 AND date_added <= 1736835350 )   )
I/PhotoManager(18617): cursor count: 57661
I/flutter (18617): query time 406

the query time is from 481ms up to 406ms

Can I remove 'date_added >= 0 AND date_added <= 1736835350' and ' duration >=0 AND duration <=86400000'
I think the performance should be better with simpler conditions

2. The cursor count

i found the cursor count is always : 57661

57661 is all my photo count, I think this is a problem

How do I set the limit to make the cursor count is my pageSize (1000)

The relevant code is here, you dont' use the limit so the cursor count is always is all my photos

    private fun cursorWithRange(
        cursor: Cursor,
        start: Int,
        pageSize: Int,
        block: (cursor: Cursor) -> Unit
    ) {
        if (!isQStorageLegacy) {
            cursor.moveToPosition(start - 1)
        }
        for (i in 0 until pageSize) {
            if (cursor.moveToNext()) {
                block(cursor)
            }
        }
    }

disable the log , the query time will up to "350ms"

2025-01-14 15:05:24.821 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 351
2025-01-14 15:05:25.172 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 350
2025-01-14 15:05:25.526 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 353
2025-01-14 15:05:25.845 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 318
2025-01-14 15:05:26.194 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 348
2025-01-14 15:05:26.596 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 401
2025-01-14 15:05:26.958 24944-25025 flutter                 com...candies.photo_manager_example  I  query time 362

3. check the file exist

Checking if the file exists is too expensive

https://github.com/fluttercandies/flutter_photo_manager/blob/main/android/src/main/kotlin/com/fluttercandies/photo_manager/core/utils/AndroidQDBUtils.kt#L206

How do I set the "checkIfExists = false“ in toAssetEntity method. In my use case, I don't need to check if the file exists here. I can check it when I use the file later.

change the sourc code set checkIfExists = false the query time will up to 150ms

25-01-14 15:10:15.698 25897-25937 flutter                    com...candies.photo_manager_example  I  query time 197
2025-01-14 15:10:15.824 25897-25937 flutter                 com...candies.photo_manager_example  I  query time 126
2025-01-14 15:10:15.988 25897-25937 flutter                 com...candies.photo_manager_example  I  query time 163
2025-01-14 15:10:16.145 25897-25937 flutter                 com...candies.photo_manager_example  I  query time 156
2025-01-14 15:10:16.299 25897-25937 flutter                 com...candies.photo_manager_example  I  query time 154
2025-01-14 15:10:16.535 25897-25937 flutter                 com...candies.photo_manager_example  I  query time 234

@AlexV525
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants