Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][retrieve] standalone Milvus panic in concurrent(=100) query retrieve varchar(length=100) scene #38731

Open
1 task done
wangting0128 opened this issue Dec 25, 2024 · 0 comments
Open
1 task done

[Bug]: [benchmark][retrieve] standalone Milvus panic in concurrent(=100) query retrieve varchar(length=100) scene #38731

wangting0128 opened this issue Dec 25, 2024 · 0 comments
Assignees
@sunby
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.1

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Dec 25, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:sunby-no_unlink-e76862d-20241217
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-9n89w

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-unlink-1-etcd-0                                           1/1     Running                  0               44h     10.104.18.54    4am-node25   <none>           <none>
wt-test-unlink-1-milvus-standalone-56b457d5f5-wt2x2               1/1     Running                  3 (12h ago)     44h     10.104.26.153   4am-node32   <none>           <none>
wt-test-unlink-1-minio-b964b9cbf-rpkpk                            1/1     Running                  0               44h     10.104.18.53    4am-node25   <none>           <none>
截屏2024-12-25 11 33 42

panic logs:

[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070079746180] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069945200271] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069156933482] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069612036020] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069560779528] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070073339238] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070150222579] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069913165532] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070252733648] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070201478102] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069676107120] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069906758589] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069304295705] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069682514217] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069291481477] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069599221722] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069970828067] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069329924097] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069233818333] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070002862819] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069515929789] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069714549708] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069105677191] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[454803069067241171]"] [oldVersion=1735052405813399982] [newVersion=1735052415813767711] [growingSegmentNum=0] [sealedSegmentNum=188]
[2024/12/24 15:00:25.812 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=454803069067241170]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [meta/target_manager.go:106] ["start to update current target for collection"] [collectionID=454803069067241170]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [meta/target_manager.go:130] ["finish to update current target for collection"] [collectionID=454803069067241170] [segments="[454803069458265969,454803069791433526,454803069535151301,454803069361959643,454803069772212597,454803069496708165,454803070054118408,454803069265853083,454803069701735536,454803069285074351,454803070137408681,454803069740177788,454803070047711461,454803069644071661,454803069240225227,454803069785026553,454803069586407511,454803069746584722,454803069919572481,454803069131305277,454803070009269776,454803069541558452,454803069509322613,454803069759398647,454803069714549708,454803069394195486,454803069951607218,454803070086153121,454803069554372542,454803070022083691,454803069547965530,454803069221004309,454803070259140594,454803069663292934,454803069733770796,454803069842689155,454803069272260161,454803070073339238,454803069407009616,454803069958014167,454803070092560057,454803069278667240,454803070066932297,454803069727363777,454803069169747650,454803069432637868,454803069932386377,454803070268554016,454803069983641983,454803069368366794,454803069823468329,454803069490301073,454803070233512808,454803069445451944,454803069099270190,454803069086456189,454803069246632109,454803069387788339,454803069336331192,454803070002862819,454803069503115352,454803069477486998,454803069612036020,454803069214597230,454803069964421113,454803069253039010,454803069708142629,454803070028490636,454803070111780873,454803069881130783,454803069310702815,454803070239919760,454803069342738302,454803070227105868,454803069144119317,454803069304295705,454803070214291983,454803070246326701,454803069483894006,454803069893944691,454803069182561794,454803069669700036,454803069092863188,454803069874723841,454803069631257373,454803069451858958,454803069291481477,454803069945200271,454803069567186516,454803070195071179,454803069188968859,454803069580000472,454803069624850259,454803070265547546,454803069605628826,454803070220698929,454803069156933482,454803069913165532,454803069829875270,454803069259446029,454803070163036448,454803069163340560,454803069124898265,454803069105677191,454803069528744074,454803070124594776,454803069073642201,454803070182257290,454803070175850341,454803069650478757,454803070201478102,454803069381381163,454803070143815628,454803069695328438,454803070079746180,454803069938793327,454803069977235026,454803069849096082,454803069317109891,454803069150526383,454803070034897583,454803069137712288,454803069419823657,454803069208190166,454803070150222579,454803069323516988,454803069195375964,454803070015676733,454803070131001737,454803069560779528,454803069112084217,454803069439044903,454803069861909980,454803069688921321,454803069297888583,454803069201783059,454803069900351640,454803069810654417,454803069797840486,454803070207885039,454803069656885836,454803069720956758,454803069515929789,454803069592814599,454803069329924097,454803070060525355,454803070156629503,454803069426230676,454803069682514217,454803069804247462,454803070118187824,454803070041304522,454803070105373933,454803069413416639,454803069522336793,454803069778619572,454803070188664238,454803069599221722,454803069176154721,454803069400602573,454803070098966994,454803069573593499,454803069080049195,454803069676107120,454803069118491242,454803069887537737,454803069471079995,454803069374973981,454803069464672969,454803069906758589,454803069765805621,454803069355552538,454803069990048927,454803069996455873,454803069752991678,454803069868316892,454803070252733648,454803069618443135,454803069227411367,454803069637664509,454803069925979430,454803069233818333,454803069855503030,454803069836282209,454803070169443392,454803069970828067,454803069817061372,454803069349145425]"] [channels="[by-dev-rootcoord-dml_1_454803069067241170v0]"] [version=1735052415813767711] [partStatsVersion="partitionStats:by-dev-rootcoord-dml_1_454803069067241170v0:[454803069067241171:0,],"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/services.go:859] ["get recovery info request received"] [traceID=053fd2194be6baddbd5b6ee232b4e6c9] [collectionID=454803069067241170] [partitionIDs="[]"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454803069067241170] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] [numOfSegments=241] ["result flushed"=188] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454803069067241171\":0}"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] [posTs=454833573753257986] [posTime=2024/12/24 14:59:50.111 +00:00]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/services.go:874] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=053fd2194be6baddbd5b6ee232b4e6c9] [collectionID=454803069067241170] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] ["# of unflushed segments"=0] ["# of flushed segments"=188] ["# of dropped segments"=53] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2024/12/24 15:00:25.813 +00:00] [DEBUG] [meta/target_manager.go:201] ["finish to update next targets for collection"] [collectionID=454803069067241170] [PartitionIDs="[454803069067241171]"]
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/segcore/segment_c.cpp:225 pc=0x7f32ee7d3a0f
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:181 pc=0x7f32ee7d3a0f
makeTryWithNoUnwrap<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:254 pc=0x7f32ee7d3a0f
makeTryWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7f32ee7d3a0f
setWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/SharedPromise-inl.h:81 pc=0x7f32ee7d3a0f
operator()<folly::Try<folly::Unit> >
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:187 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:941 pc=0x7f32ee7d3a0f
invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:139 pc=0x7f32ee7d3a0f
invoke<folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >, folly::Unit>
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:341 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:388 pc=0x7f32ee7d3a0f
makeTryWithNoUnwrap<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:265 pc=0x7f32ee7d3a0f
makeTryWith<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:387 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/detail/Core.h:629 pc=0x7f32ee7d3a0f
callBig<folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >(folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Function.h:359 pc=0x7f32ee7d3a0f
_ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEEclES6_SA_SC_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7f32e9f52d3f
operator()
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/futures/detail/Core.cpp:616 pc=0x7f32e9f52d3f
operator()
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:187 pc=0x7f32e9f52d3f
callSmall<folly::Executor::KeepAlive<>::add<folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)> >(folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)>&&) &&::<lambda()> >
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7f32e9f52d3f
_ZN5folly6detail8function14FunctionTraitsIFvvEEclEv
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7f32e9e72e75
_ZN5folly15catch_exceptionIRNS_8FunctionIFvvEEERDoFvPKcEJRS6_EvEET2_OT_OT0_DpOT1_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/lang/Exception.h:286 pc=0x7f32e9e72e75
_ZN5folly8Executor18invokeCatchingExnsINS_8FunctionIFvvEEEEEvPKcT_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:234 pc=0x7f32e9e72e75
_ZN5folly18ThreadPoolExecutor7runTaskERKSt10shared_ptrINS0_6ThreadEEONS0_4TaskE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/ThreadPoolExecutor.cpp:102 pc=0x7f32e9e72e75
_ZN5folly21CPUThreadPoolExecutor9threadRunESt10shared_ptrINS_18ThreadPoolExecutor6ThreadEE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/CPUThreadPoolExecutor.cpp:333 pc=0x7f32e9e35e74
_ZSt13__invoke_implIvRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEERPS1_JRS4_EET_St21__invoke_memfun_derefOT0_OT1_DpOT2_
	/usr/include/c++/12/bits/invoke.h:74 pc=0x7f32e9e7544b
_ZSt8__invokeIRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEJRPS1_RS4_EENSt15__invoke_resultIT_JDpT0_EE4typeEOSC_DpOSD_
	/usr/include/c++/12/bits/invoke.h:96 pc=0x7f32e9e7544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
	/usr/include/c++/12/functional:495 pc=0x7f32e9e7544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EEclIJEvEET0_DpOT_
	/usr/include/c++/12/functional:580 pc=0x7f32e9e7544b
_ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallISt5_BindIFMNS_18ThreadPoolExecutorEFvSt10shared_ptrINS7_6ThreadEEEPS7_SA_EEEEvRNS1_4DataE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7f32e9e7544b
(null)
	(null):0 pc=0x7f32ebc1c252
(null)
	(null):0 pc=0x7f32ebe00ac2
(null)
	(null):0 pc=0x7f32ebe9284f
(null)
	(null):0 pc=0xffffffffffffffff

client error log: 2024-12-24 15:00:26,770 ~ 2024-12-24 15:09:08,556

Expected Behavior

No response

Steps To Reproduce

1. create a collection with fields: 'id'(INT64, primary key), 'float_vector'(3dim), 'varchar_100'
2. build FLAT index on field float_vector
3. insert 900m data
4. flush collection
5. load collection
6. concurrent requests query retrieve varchar_100 <- concurrent 100, and milvus panic

Milvus Log

No response

Anything else?

test result:

{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'wt-test-unlink-1-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 3,
                                                    'max_length': 101,
                                                    'varchar_filled': True,
                                                    'dataset_name': 'local',
                                                    'dataset_size': '900m',
                                                    'ni_per': 50000},
                                 'collection_params': {'other_fields': ['varchar_100'], 'shards_num': 1, 'collection_name': 'scene_varchar_100'},
                                 'index_params': {'index_type': 'FLAT', 'index_param': {}},
                                 'concurrent_params': {'concurrent_number': 100, 'during_time': '30m', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'output_fields': ['varchar_100'],
                                                                  'limit': 10,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'random_count': 10,
                                                                  'random_range': [0, 900000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024122424787695,
            'datetime': '2024-12-24 03:54:38.131492',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 19.6642},
                            'insert': {'total_time': 2382.3471, 'VPS': 377778.704, 'batch_time': 0.1324, 'batch': 50000},
                            'flush': {'RT': 3.0173},
                            'load': {'RT': 469.0147},
                            'Locust': {'Aggregated': {'Requests': 2421,
                                                      'Fails': 1211,
                                                      'RPS': 1.35,
                                                      'fail_s': 0.5,
                                                      'RT_max': 189903.26,
                                                      'RT_avg': 72074.93,
                                                      'TP50': 101000.0,
                                                      'TP99': 188000.0},
                                       'query': {'Requests': 2421,
                                                 'Fails': 1211,
                                                 'RPS': 1.35,
                                                 'fail_s': 0.5,
                                                 'RT_max': 189903.26,
                                                 'RT_avg': 72074.93,
                                                 'TP50': 101000.0,
                                                 'TP99': 188000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 25, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 25, 2024
@yanliang567 yanliang567 added this to the 2.5.1 milestone Dec 25, 2024
@yanliang567 yanliang567 removed their assignment Dec 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunby sunby

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.1
Development

No branches or pull requests
3 participants
@sunby @wangting0128 @yanliang567