Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][standalone] hybrid_search a released partition raises error collection not found which expected error is collection not loaded in a dml & multi-partition scene #38586

Open
1 task done
wangting0128 opened this issue Dec 19, 2024 · 3 comments
Open
1 task done

[Bug]: [benchmark][standalone] hybrid_search a released partition raises error collection not found which expected error is collection not loaded in a dml & multi-partition scene #38586

wangting0128 opened this issue Dec 19, 2024 · 3 comments
Assignees
@wangting0128 @xiaocai2333
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.1

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Dec 19, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20241218-78438ef4-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq   
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: bitmap-corn-1734577200
test case name: test_bitmap_locust_dml_partitions_standalone

server:

NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-corn-1734577200-6-etcd-0                                   1/1     Running     0               4h31m   10.104.17.201   4am-node23   <none>           <none>
bitmap-corn-1734577200-6-milvus-standalone-74689b8bd7-czqb8       1/1     Running     1 (4h31m ago)   4h31m   10.104.21.242   4am-node24   <none>           <none>
bitmap-corn-1734577200-6-minio-5fdd9f768d-vkgmv                   1/1     Running     0               4h31m   10.104.17.204   4am-node23   <none>           <none>

{pod=~"bitmap-corn-1734577200-6-milvus-standalone-74689b8bd7-czqb8"} |= "2d6d4fa0ef0456fd1b286f44c6660d5a"
hybrid_search_failed.log

client log:

, 'scene_test_partition_hybrid_search_G4TmAm3n', ''], kwargs: {}, [requestId: a11ace8c-bdda-11ef-9af1-3ab57b090b4f] (api_request.py:77)
[2024-12-19 07:26:57,530 - DEBUG - fouram]: (api_response) : [Partition] {"name":"scene_test_partition_hybrid_search_G4TmAm3n","collection_name":"fouram_Xzj68qHW","description":""}, [requestId: a11ace8c-bdda-11ef-9af1-3ab57b090b4f] (api_request.py:44)
[2024-12-19 07:26:57,530 - DEBUG - fouram]: [Base] Create partition scene_test_partition_hybrid_search_G4TmAm3n of collection(fouram_Xzj68qHW) (base.py:818)
[2024-12-19 07:26:57,654 - DEBUG - fouram]: (api_request)  : [Collection.insert] args: <Collection.insert fields: 15, length: 3000, content: [ [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'list'>, dtype<>` [0.6622176546742689, 0.7 ... ], [ `type<class 'list'>, dtype<>` [0.9501366174321724, 0.5 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'str'>, dtype<>` 0 ... ], [ `type<class 'bool'>, dtype<>` False ... ], [ `type<class 'list'>, dtype<>` [0, 0, 0, 0, 0, 0, 0, 0, ... ], [ `type<class 'list'>, dtype<>` [0, 0, 0, 0, 0, 0, 0, 0, ... ], [ `type<class 'list'>, dtype<>` [0, 0, 0, 0, 0, 0, 0, 0, ... ], [ `type<class 'list'>, dtype<>` [0, 0, 0, 0, 0, 0, 0, 0, ... ], [ `type<class 'list'>, dtype<>` ['0', '0', '0', '0', '0' ... ], [ `type<class 'list'>, dtype<>` [False, False, False, Fa ... ] ]>, ['scene_test_partition_hybrid_search_G4TmAm3n'], kwargs: {'timeout': 600}, [requestId: a12f8962-bdda-11ef-9af1-3ab57b090b4f] (api_request.py:77)
[2024-12-19 07:26:58,027 - DEBUG - fouram]: [Base] Start flush partition scene_test_partition_hybrid_search_G4TmAm3n, kwargs: {'timeout': 600} (base.py:833)
[2024-12-19 07:27:56,566 - DEBUG - fouram]: [Base] Partition scene_test_partition_hybrid_search_G4TmAm3n num entities: (3000) (base.py:829)
[2024-12-19 07:27:56,800 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:float_vector_1, index_params:{'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}, 'index_type': 'HNSW'} (base.py:1123)
[2024-12-19 07:30:49,006 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:int32_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:30:57,048 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:int64_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:30:58,135 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_int8_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:30:59,262 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:float_vector, index_params:{'index_type': 'IVF_SQ8', 'metric_type': 'L2', 'params': {'nlist': 1024}} (base.py:1123)
[2024-12-19 07:31:01,085 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:int8_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:02,410 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:varchar_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:03,513 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_int16_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:04,494 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_bool_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:05,518 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_varchar_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:06,548 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:int16_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:07,614 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:bool_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:09,287 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_int32_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:10,323 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_G4TmAm3n start building index, field_name:array_int64_1, index_params:{'index_type': 'BITMAP'} (base.py:1123)
[2024-12-19 07:31:11,275 - DEBUG - fouram]: [Base] Start load partition scene_test_partition_hybrid_search_G4TmAm3n, replica_number:1, kwargs:{'timeout': 600} (base.py:839)
[2024-12-19 07:31:17,345 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_G4TmAm3n hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_response'} (base.py:868)
[2024-12-19 07:31:17,518 - DEBUG - fouram]: [Base] Start release partition scene_test_partition_hybrid_search_G4TmAm3n (base.py:845)
[2024-12-19 07:31:21,129 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_G4TmAm3n hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_error_response', 'check_items': {'code': 65535, 'message': 'not loaded'}} (base.py:868)
[2024-12-19 07:31:21,129 - DEBUG - fouram]: (api_request)  : [Partition.hybrid_search] args: [[<pymilvus.client.abstract.AnnSearchRequest object at 0x7f7d57511760>, <pymilvus.client.abstract.AnnSearchRequest object at 0x7f7d57511790>], <pymilvus.client.abstract.RRFRanker object at 0x7f7d59c0c790>, 1, ['*'], 600, -1], kwargs: {}, [requestId: 3e3aa35e-bddb-11ef-9af1-3ab57b090b4f] (api_request.py:77)
[2024-12-19 07:31:22,008 - ERROR - fouram]: (api_response) : [Partition.hybrid_search] <MilvusException: (code=100, message=fail to search on QueryNode 1: collection not found[collection=454709062265209954])>, [requestId: 3e3aa35e-bddb-11ef-9af1-3ab57b090b4f] (api_request.py:57)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`, divided into 10 partitions
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. load and search partial partitions & DML(upsert) requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'float_vector_1': 768dim
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                HNSW: 'float_vector_1'

                BITMAP: all scalar fields
            3. insert 5 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - scene_insert_partition
                    (partition: create->insert->flush->release->drop)
                - scene_test_partition
                    (partition: create->insert->flush->index again->load->search->release->search failed->drop)
                - scene_test_partition_hybrid_search
                    (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop) <- hybrid_search raises not expected error
                - release_partitions: 10 prepared partitions
                - upsert: batch=1

Milvus Log

No response

Anything else?

test result:

[2024-12-19 07:34:16,862 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: grpc     release_partitions                                                               147     0(0.00%) |   5158      13  103289    390 |    0.01        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: grpc     scene_insert_partition                                                           163     0(0.00%) |  41169    3581  136316  27000 |    0.02        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: grpc     scene_test_partition                                                             147     0(0.00%) | 330098   50002  598220 311000 |    0.01        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: grpc     scene_test_partition_hybrid_search                                               148     1(0.68%) | 340593   30427  740004 324000 |    0.01        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: grpc     upsert                                                                           141     0(0.00%) |     38       3     631      7 |    0.01        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]:          Aggregated                                                                       746     1(0.13%) | 142636       3  740004  30000 |    0.07        0.00 (stats.py:789)
[2024-12-19 07:34:16,862 -  INFO - fouram]:  (stats.py:790)
[2024-12-19 07:34:16,867 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_16c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16.0', 'memory': '16Gi'}, 'requests': {'cpu': '9.0', 'memory': '9Gi'}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
                       'pulsarv3': {'enabled': False},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20241218-78438ef4-amd64'}}},
            'host': 'bitmap-corn-1734577200-6-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_bitmap_locust_dml_partitions_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'max_length': 100,
                                                    'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
                                                                      'int16_1': {'index_type': 'BITMAP'},
                                                                      'int32_1': {'index_type': 'BITMAP'},
                                                                      'int64_1': {'index_type': 'BITMAP'},
                                                                      'varchar_1': {'index_type': 'BITMAP'},
                                                                      'bool_1': {'index_type': 'BITMAP'},
                                                                      'array_int8_1': {'index_type': 'BITMAP'},
                                                                      'array_int16_1': {'index_type': 'BITMAP'},
                                                                      'array_int32_1': {'index_type': 'BITMAP'},
                                                                      'array_int64_1': {'index_type': 'BITMAP'},
                                                                      'array_varchar_1': {'index_type': 'BITMAP'},
                                                                      'array_bool_1': {'index_type': 'BITMAP'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
                                                                                        'other_params': {'dataset': 'random_algorithm',
                                                                                                         'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                              'specify_range': [-128, 128],
                                                                                                                              'max_capacity': 13}}},
                                                                       'array_int16_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                               'specify_range': [-200, 200],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int32_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                               'specify_range': [-300, 300],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int64_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 13},
                                                                                           'other_params': {'dataset': 'random_algorithm',
                                                                                                            'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                                 'specify_range': [-1500, 1500],
                                                                                                                                 'max_capacity': 13}}},
                                                                       'array_bool_1': {'params': {'max_capacity': 13}},
                                                                       'int8_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                   'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                        'specify_range': [-128, 128],
                                                                                                                        'max_capacity': 13}}},
                                                                       'int16_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                         'specify_range': [-200, 200],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int32_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                         'specify_range': [-300, 300],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int64_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                         'specify_range': [-400, 432],
                                                                                                                         'batch': 50,
                                                                                                                         'max_capacity': 13}}},
                                                                       'varchar_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                      'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                           'specify_range': [-1500, 1500],
                                                                                                                           'max_capacity': 13}}}},
                                                    'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                        'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                        'partition_9'],
                                                                         'data_repeated': False},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 5000000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
                                                                        'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
                                                                        'array_bool_1'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 10, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_insert_partition',
                                                       'weight': 1,
                                                       'params': {'data_size': 3000, 'ni': 1000, 'with_flush': True, 'timeout': 600}},
                                                      {'type': 'scene_test_partition',
                                                       'weight': 1,
                                                       'params': {'data_size': 3000,
                                                                  'ni': 3000,
                                                                  'nq': 1,
                                                                  'search_param': {'nprobe': 64},
                                                                  'limit': 1,
                                                                  'expr': None,
                                                                  'output_fields': ['*'],
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 600,
                                                                  'search_counts': 1}},
                                                      {'type': 'scene_test_partition_hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'hybrid_search_counts': 1,
                                                                  'data_size': 3000,
                                                                  'ni': 3000}},
                                                      {'type': 'release_partitions',
                                                       'weight': 1,
                                                       'params': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                 'partition_5', 'partition_6', 'partition_7', 'partition_8', 'partition_9'],
                                                                  'timeout': 180,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'upsert',
                                                       'weight': 1,
                                                       'params': {'nb': 1,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 5000000,
                                                                  'shuffle_id': False,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024121973902972,
            'datetime': '2024-12-19 03:03:10.100268',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 4531.9779,
                                      'float_vector_1': {'RT': 0.5186},
                                      'int8_1': {'RT': 0.5195},
                                      'int16_1': {'RT': 0.5195},
                                      'int32_1': {'RT': 0.5192},
                                      'int64_1': {'RT': 0.517},
                                      'varchar_1': {'RT': 0.5173},
                                      'bool_1': {'RT': 0.5172},
                                      'array_int8_1': {'RT': 0.5181},
                                      'array_int16_1': {'RT': 0.517},
                                      'array_int32_1': {'RT': 0.5193},
                                      'array_int64_1': {'RT': 0.5203},
                                      'array_varchar_1': {'RT': 0.5234},
                                      'array_bool_1': {'RT': 0.5227}},
                            'insert': {'total_time': 513.2447, 'VPS': 9753.9767, 'batch_time': 0.5132, 'batch': 5000.0},
                            'flush': {'RT': 10.6024},
                            'load': {'RT': 7.0097},
                            'Locust': {'Aggregated': {'Requests': 746,
                                                      'Fails': 1,
                                                      'RPS': 0.07,
                                                      'fail_s': 0.0,
                                                      'RT_max': 740004.88,
                                                      'RT_avg': 142636.4,
                                                      'TP50': 30000.0,
                                                      'TP99': 553000.0},
                                       'release_partitions': {'Requests': 147,
                                                              'Fails': 0,
                                                              'RPS': 0.01,
                                                              'fail_s': 0.0,
                                                              'RT_max': 103289.44,
                                                              'RT_avg': 5158.87,
                                                              'TP50': 390.0,
                                                              'TP99': 87000.0},
                                       'scene_insert_partition': {'Requests': 163,
                                                                  'Fails': 0,
                                                                  'RPS': 0.02,
                                                                  'fail_s': 0.0,
                                                                  'RT_max': 136316.51,
                                                                  'RT_avg': 41169.48,
                                                                  'TP50': 27000.0,
                                                                  'TP99': 133000.0},
                                       'scene_test_partition': {'Requests': 147,
                                                                'Fails': 0,
                                                                'RPS': 0.01,
                                                                'fail_s': 0.0,
                                                                'RT_max': 598220.6,
                                                                'RT_avg': 330098.98,
                                                                'TP50': 311000.0,
                                                                'TP99': 591000.0},
                                       'scene_test_partition_hybrid_search': {'Requests': 148,
                                                                              'Fails': 1,
                                                                              'RPS': 0.01,
                                                                              'fail_s': 0.01,
                                                                              'RT_max': 740004.88,
                                                                              'RT_avg': 340593.35,
                                                                              'TP50': 325000.0,
                                                                              'TP99': 609000.0},
                                       'upsert': {'Requests': 141,
                                                  'Fails': 0,
                                                  'RPS': 0.01,
                                                  'fail_s': 0.0,
                                                  'RT_max': 631.25,
                                                  'RT_avg': 38.36,
                                                  'TP50': 7,
                                                  'TP99': 460.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Dec 19, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Dec 19, 2024
@yanliang567
Copy link
Contributor

/assign @xiaocai2333
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 19, 2024
@xiaocai2333
Copy link
Contributor

When a collection contains only one partition, releasing that partition will trigger the release of the entire collection. At this point, performing a query will result in an error: collection not found.

2024-12-19 15:31:17.595	
[2024/12/19 07:31:17.595 +00:00] [DEBUG] [proxy/impl.go:1692] ["ReleasePartitions received"] [traceID=98b70623c39242d15524b787d8192300] [role=proxy] [db=default] [collection=fouram_Xzj68qHW] [partitions="[scene_test_partition_1x3qQuW3]"]
2024-12-19 15:31:17.742	
[2024/12/19 07:31:17.742 +00:00] [DEBUG] [proxy/impl.go:1705] ["ReleasePartitions enqueued"] [traceID=98b70623c39242d15524b787d8192300] [role=proxy] [db=default] [collection=fouram_Xzj68qHW] [partitions="[scene_test_partition_1x3qQuW3]"] [BeginTS=454713272619237380] [EndTS=454713272619237380]
2024-12-19 15:31:17.846	
[2024/12/19 07:31:17.845 +00:00] [DEBUG] [proxy/meta_cache.go:641] ["Reload collection from root coordinator "] [traceID=98b70623c39242d15524b787d8192300] [collectionName=fouram_Xzj68qHW] ["time (milliseconds) take "=103]
2024-12-19 15:31:17.850	
[2024/12/19 07:31:17.850 +00:00] [INFO] [querycoordv2/services.go:427] ["release partitions"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954] [partitions="[454709062463585047]"]
2024-12-19 15:31:21.009	
[2024/12/19 07:31:21.009 +00:00] [INFO] [job/scheduler.go:150] ["start to pre-execute job"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954]
2024-12-19 15:31:21.009	
[2024/12/19 07:31:21.009 +00:00] [INFO] [job/scheduler.go:158] ["start to execute job"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954]
2024-12-19 15:31:21.009	
[2024/12/19 07:31:21.009 +00:00] [INFO] [job/job_release.go:174] ["release partitions covers all partitions, will remove the whole collection"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954] [partitionIDs="[454709062463585047]"]
2024-12-19 15:31:21.011	
[2024/12/19 07:31:21.011 +00:00] [INFO] [proxy/impl.go:122] ["received request to invalidate collection meta cache"] [traceID=98b70623c39242d15524b787d8192300] [module=Proxy] [role=proxy] [db=] [collectionName=] [collectionID=454709062265209954] [msgType=ReleaseCollection] [partitionName=]
2024-12-19 15:31:21.011	
[2024/12/19 07:31:21.011 +00:00] [DEBUG] [proxy/meta_cache.go:884] ["remove collection by id"] [traceID=98b70623c39242d15524b787d8192300] [module=Proxy] [id=454709062265209954] [collection="[fouram_Xzj68qHW]"] [currentVersion=454713273211420673] [version=0] [removeVersion=false]
2024-12-19 15:31:21.011	
[2024/12/19 07:31:21.011 +00:00] [INFO] [proxy/impl.go:151] ["complete to invalidate collection meta cache"] [traceID=98b70623c39242d15524b787d8192300] [module=Proxy] [role=proxy] [db=] [collectionName=] [collectionID=454709062265209954] [msgType=ReleaseCollection] [partitionName=] [type=ReleaseCollection]
2024-12-19 15:31:21.011	
[2024/12/19 07:31:21.011 +00:00] [INFO] [proxy/impl.go:205] ["complete to invalidate collection meta cache"] [traceID=98b70623c39242d15524b787d8192300] [module=Proxy] [role=proxy] [db=] [collectionName=] [collectionID=454709062265209954] [msgType=ReleaseCollection] [partitionName=]
2024-12-19 15:31:21.413	
[2024/12/19 07:31:21.413 +00:00] [INFO] [job/scheduler.go:144] ["start to post-execute job"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954]
2024-12-19 15:31:21.413	
[2024/12/19 07:31:21.413 +00:00] [INFO] [job/scheduler.go:146] ["job finished"] [traceID=98b70623c39242d15524b787d8192300] [collectionID=454709062265209954]
2024-12-19 15:31:21.414	
[2024/12/19 07:31:21.414 +00:00] [DEBUG] [proxy/impl.go:1723] ["ReleasePartitions done"] [traceID=98b70623c39242d15524b787d8192300] [role=proxy] [db=default] [collection=fouram_Xzj68qHW] [partitions="[scene_test_partition_1x3qQuW3]"] [BeginTS=454713272619237380] [EndTS=454713272619237380]

BTW, the query request occurs after the release request, with no load operation in between. @wangting0128
image
/assign @wangting0128

@xiaocai2333
Copy link
Contributor

maybe the error log is not user-friendly.

@xiaocai2333 xiaocai2333 mentioned this issue Dec 19, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Dec 23, 2024
@xiaocai2333
enhance: Return collection not loaded rather than not found on queryn…
bb5f38e 
…ode (#38593)

issue: #38586

---------

Signed-off-by: Cai Zhang <[email protected]>
@yanliang567 yanliang567 modified the milestones: 2.5.0, 2.5.1 Dec 24, 2024
NicoYuan1986 pushed a commit to NicoYuan1986/milvus that referenced this issue Dec 26, 2024
@xiaocai2333 @NicoYuan1986
enhance: Return collection not loaded rather than not found on queryn…
9b96feb 
…ode (milvus-io#38593)

issue: milvus-io#38586

---------

Signed-off-by: Cai Zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangting0128 wangting0128

@xiaocai2333 xiaocai2333

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.1
Development

No branches or pull requests
3 participants
@wangting0128 @xiaocai2333 @yanliang567