how to debug c++ code in milvus

[cherrylzhao]

hi, everyone
I have modified the mode to "Debug" in Makefile, then compiled milvus 2.4.18 in centos 7.9 successfully.
currently, Attu can connect to my local compiled milvus binary.
so I want to investigate the vector query process internal, I try to attach the pid of milvus in gdb,
but it seems that breakpoint in c++ code could not stop, please give me some hint of how can I correctly debug c++ code.

thanks

[yhmo]

Milvus process is running with a lot of parallel threads, if you set a breakpoint to a thread, the other thread might encounter timeout error and run into a weird state. So, milvus developers don't "attach and debug", they add temp logs in source code and print the runtime variables to diagnose, or review the full log to investigate.

[cherrylzhao]

thanks to reply !
currently, I just want to know the invoke chain of query process in source code, could you please give me some sample debug logs of that?

[yhmo]

All requests are received by proxy node, the search request entry is here:

milvus/internal/proxy/impl.go

Line 2991 in bb5f38e

func (node *Proxy) Search(ctx context.Context, request *milvuspb.SearchRequest) (*milvuspb.SearchResults, error) {

Once a search request is received, a searchTask is put into a scheduler to queue:
https://github.com/milvus-io/milvus/blob/master/internal/proxy/task_scheduler.go
https://github.com/milvus-io/milvus/blob/master/internal/proxy/task_search.go

searchTask.searchShard() is called if the searchTask is to be executed:

milvus/internal/proxy/task_search.go

Line 767 in bb5f38e

func (t *searchTask) searchShard(ctx context.Context, nodeID int64, qn types.QueryNodeClient, channel string) error {

A query node's Search() is called by searchTask.searchShard():

milvus/internal/querynodev2/services.go

Line 740 in bb5f38e

func (node *QueryNode) Search(ctx context.Context, req *querypb.SearchRequest) (*internalpb.SearchResults, error) {

ShardDelegator.Serch() is called. Each shard delegator is a leader node for a shard of a collection. By default, a collection's shard_number is 1, there is only one shard delegator. If shard_number is N, there will be N shard delegators work for the search request.

milvus/internal/querynodev2/delegator/delegator.go

Line 273 in 82d85e8

func (sd *shardDelegator) search(ctx context.Context, req *querypb.SearchRequest, sealed []SnapshotItem, growing []SegmentEntry) ([]*internalpb.SearchResults, error) {

remoteWorker.SearchSegments() is called by the shardDelegator.search()

milvus/internal/querynodev2/cluster/worker.go

Line 191 in 82d85e8

func (w *remoteWorker) SearchSegments(ctx context.Context, req *querypb.SearchRequest) (*internalpb.SearchResults, error) {

QueryNode.SearchSegments() is called by emoteWorker.SearchSegments()

milvus/internal/querynodev2/services.go

Line 655 in 82d85e8

func (node *QueryNode) SearchSegments(ctx context.Context, req *querypb.SearchRequest) (*internalpb.SearchResults, error) {

SearchTask is created by QueryNode.SearchSegments() and executed

milvus/internal/querynodev2/tasks/search_task.go

Line 133 in 82d85e8

func (t *SearchTask) Execute() error {

Segment.search() is called by searchSegments()

milvus/internal/querynodev2/segments/search.go

Line 48 in 82d85e8

searchResult, err := s.Search(ctx, searchReq)

The implementation of Segment.search(), from this method, you will see the code runs into CGo
https://github.com/milvus-io/milvus/blob/82d85e8c57b8c144e53789fe725960d5c0600a86/internal/querynodev2/segments/segment.go#L492C10-L492C22

The search entry in CGo:

milvus/internal/core/src/segcore/segment_c.cpp

Line 110 in 82d85e8

AsyncSearch(CTraceContext c_trace,

[cherrylzhao]

thanks for your detail explain, that help me a lot !