enhance: record memory size (uncompressed) item for index #38770

chyezh · 2024-12-26T03:57:15Z

Current milvus use a serialized index size(compressed) for estimate resource for loading.
Add a new field MemSize (before compressing) for index to estimate resource.

sre-ci-robot · 2024-12-26T03:57:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign yanliang567 after the PR has been reviewed.
You can assign the PR to them by writing /assign @yanliang567 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-12-26T04:48:36Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T04:50:52Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T05:47:17Z

rerun cpp-unit-test

mergify · 2024-12-26T06:10:10Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T08:33:46Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:07:20Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:32:42Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:40:59Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T13:30:49Z

rerun cpp-unit-test

mergify · 2024-12-26T14:19:25Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

codecov · 2024-12-26T15:28:38Z

Codecov Report

Attention: Patch coverage is 68.90244% with 51 lines in your changes missing coverage. Please review.

Project coverage is 81.13%. Comparing base (dd81a13) to head (dbf4b36).
Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/index/CreateIndexResult.cpp	68.75%	10 Missing ⚠️
internal/util/indexcgowrapper/index.go	37.50%	9 Missing and 1 partial ⚠️
internal/core/src/index/TextMatchIndex.cpp	0.00%	8 Missing ⚠️
internal/core/src/common/protobuf_utils.cpp	50.00%	6 Missing ⚠️
internal/core/src/indexbuilder/index_c.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/StringIndexMarisa.cpp	0.00%	5 Missing ⚠️
internal/core/src/index/ScalarIndexSort.cpp	0.00%	4 Missing ⚠️
internal/core/src/common/protobuf_utils.h	83.33%	1 Missing ⚠️
internal/core/src/storage/DiskFileManagerImpl.cpp	50.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #38770    +/-   ##
========================================
  Coverage   81.12%   81.13%            
========================================
  Files        1385     1389     +4     
  Lines      195922   196151   +229     
========================================
+ Hits       158939   159138   +199     
- Misses      31396    31422    +26     
- Partials     5587     5591     +4

Components	Coverage Δ
Client	`79.12% <ø> (ø)`
Core	`69.43% <59.40%> (-0.03%)`	⬇️
Go	`83.11% <84.12%> (+0.01%)`	⬆️

Files with missing lines	Coverage Δ
internal/core/src/index/BitmapIndex.cpp	`81.09% <100.00%> (+0.08%)`	⬆️
internal/core/src/index/BitmapIndex.h	`44.44% <ø> (ø)`
internal/core/src/index/CreateIndexResult.h	`100.00% <100.00%> (ø)`
internal/core/src/index/HybridScalarIndex.cpp	`79.12% <100.00%> (-0.33%)`	⬇️
internal/core/src/index/HybridScalarIndex.h	`39.02% <ø> (ø)`
internal/core/src/index/Index.h	`100.00% <ø> (ø)`
internal/core/src/index/InvertedIndexTantivy.cpp	`92.41% <100.00%> (-0.23%)`	⬇️
internal/core/src/index/InvertedIndexTantivy.h	`53.84% <ø> (ø)`
internal/core/src/index/ScalarIndexSort.h	`38.46% <ø> (ø)`
internal/core/src/index/StringIndexMarisa.h	`71.42% <ø> (ø)`
... and 29 more

... and 36 files with indirect coverage changes

chyezh · 2024-12-27T06:39:30Z

rerun cpp-unit-test

mergify · 2024-12-27T07:27:22Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-27T07:30:26Z

rerun cpp-unit-test

chyezh · 2024-12-27T07:30:45Z

rerun go-sdk

mergify · 2024-12-27T08:13:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-27T08:24:07Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-27T16:13:07Z

rerun go-sdk

chyezh · 2024-12-27T16:13:12Z

rerun cpp-unit-test

mergify · 2024-12-27T17:01:06Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-28T04:57:27Z

rerun cpp-unit-test

mergify · 2024-12-28T05:37:14Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-28T06:20:45Z

rerun cpp-unit-test

mergify · 2024-12-28T06:42:38Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-28T08:47:14Z

rerun cpp-unit-test

mergify · 2024-12-28T09:10:27Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-29T06:55:07Z

rerun cpp-unit-test

mergify · 2024-12-29T06:57:30Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-29T07:05:39Z

rerun cpp-unit-test

mergify · 2024-12-29T07:50:41Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-29T11:05:32Z

rerun cpp-unit-test

mergify · 2024-12-29T11:33:21Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-29T12:24:26Z

rerun cpp-unit-test

mergify · 2024-12-29T12:47:03Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

mergify · 2024-12-30T07:05:30Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T08:02:07Z

rerun cpp-unit-test

mergify · 2024-12-30T08:49:01Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T11:08:20Z

rerun cpp-unit-test

mergify · 2024-12-30T11:50:31Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T13:37:36Z

rerun cpp-unit-test

mergify · 2024-12-30T14:23:43Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-31T01:51:21Z

rerun cpp-unit-test

mergify · 2024-12-31T02:36:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Signed-off-by: chyezh <[email protected]>

mergify · 2024-12-31T10:44:13Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-31T11:03:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Signed-off-by: chyezh <[email protected]>

mergify · 2024-12-31T12:11:19Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

sre-ci-robot added area/internal-api size/XXL Denotes a PR that changes 1000+ lines. labels Dec 26, 2024

sre-ci-robot requested review from congqixia and czs007 December 26, 2024 03:57

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 26, 2024

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 4cb8035 to 1ebba3e Compare December 26, 2024 11:44

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 1ebba3e to 07910b5 Compare December 30, 2024 06:20

enhance: record memory size (uncompressed) item for index

5a89096

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 07910b5 to 5a89096 Compare December 30, 2024 06:21

chyezh mentioned this pull request Dec 30, 2024

enhance: record memory size (uncompressed) item for index #38844

Open

fix: unittest

2828203

Signed-off-by: chyezh <[email protected]>

fix: unittest

dbf4b36

Signed-off-by: chyezh <[email protected]>

enhance: record memory size (uncompressed) item for index #38770

Are you sure you want to change the base?

enhance: record memory size (uncompressed) item for index #38770

Conversation

chyezh commented Dec 26, 2024

sre-ci-robot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

codecov bot commented Dec 26, 2024 • edited Loading

Codecov Report

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 28, 2024

mergify bot commented Dec 28, 2024

chyezh commented Dec 28, 2024

mergify bot commented Dec 28, 2024

chyezh commented Dec 28, 2024

mergify bot commented Dec 28, 2024

chyezh commented Dec 29, 2024

mergify bot commented Dec 29, 2024

chyezh commented Dec 29, 2024

mergify bot commented Dec 29, 2024

chyezh commented Dec 29, 2024

mergify bot commented Dec 29, 2024

chyezh commented Dec 29, 2024

mergify bot commented Dec 29, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

codecov bot commented Dec 26, 2024 •

edited

Loading