[Bug]: [benchmark][retrieve] milvus(enabled all mmap) OOM when loading a collection with array(varchar) field

[wangting0128]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:sunby-no_unlink-ceed4f7-20241224
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-w894f

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-unlink-etcd-0                                             1/1     Running                  0               7d19h   10.104.26.150   4am-node32   <none>           <none>
wt-test-unlink-milvus-standalone-5f8b7b6b95-g64sj                 1/1     Running                  4 (15h ago)     24h     10.104.26.176   4am-node32   <none>           <none>
wt-test-unlink-minio-67b75cfbf4-dk5ht                             1/1     Running                  0               7d19h   10.104.26.152   4am-node32   <none>           <none>

client log:

Expected Behavior

No response

Steps To Reproduce

1. create a collection with fields: 'id'(INT64, primary key), 'float_vector'(3dim), 'array_varchar_1'
2. build FLAT index on field float_vector
3. insert 900m data
4. flush collection
5. load collection <- milvus OOM

Milvus Log

No response

Anything else?

client config:

{
     "dataset_params": {
          "metric_type": "L2",
          "dim": 3,
          "dataset_name": "local",
          "dataset_size": "900m",
          "ni_per": 50000,
          "scalars_params": {
               "array_varchar_1": {
                    "params": {
                         "max_length": 100,
                         "max_capacity": 10
                    }
               }
          }
     },
     "collection_params": {
          "shards_num": 1,
          "other_fields": [
               "array_varchar_1"
          ],
          "collection_name": "scene_array_varchar_1"
     },
     "index_params": {
          "index_type": "FLAT",
          "index_param": {}
     },
     "concurrent_params": {
          "concurrent_number": [
               10,
               100
          ],
          "during_time": "30m",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "",
                    "output_fields": [
                         "array_varchar_1"
                    ],
                    "limit": 10,
                    "timeout": 600,
                    "random_data": true,
                    "random_count": 10,
                    "random_range": [
                         0,
                         900000000
                    ],
                    "field_name": "id",
                    "field_type": "int64"
               }
          }
     ]
}

[MrPresent-Han]

/assign

[xiaofan-luan]

we need better array offset design

1. uint64 is probably too much
2. the offset array has to be mmaped