删除数据失败问题

[elpsycongroo]

版本：Milvus 2.4.5 k8s部署
现象：Attu或API删除数据，返回成功结果，但依旧可以查询到该条数据，且未被删除。该现象为偶发，有一定概率出现。
日志：删除成功的query node日志显示：

[2024/12/10 06:43:24.762 +00:00] [INFO] [pipeline/delete_node.go:57] ["pipeline fetch delete msg"] [collectionID=453784516644544182] [partitionID=-1] [insertRowNum=1] [timestampMin=454508676251910146] [timestampMax=454508676251910146]
[2024/12/10 06:43:24.762 +00:00] [DEBUG] [delegator/delegator_data.go:183] ["start to process delete"] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [replicaID=454028779275157507] [ts=454508676278386690]
[2024/12/10 06:43:24.762 +00:00] [DEBUG] [delegator/delegator_data.go:345] ["delegator plan to applyDelete via worker"] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [replicaID=454028779275157507] [segmentID=454029151015046147] [workerID=36] [forwardRowCount=1]
[2024/12/10 06:43:24.762 +00:00] [INFO] [querynodev2/services.go:1355] ["QueryNode received worker delete request"] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [segmentID=454029151015046147] [scope=Historical]
[2024/12/10 06:43:24.762 +00:00] [DEBUG] [querynodev2/services.go:1356] ["Worker delete detail"] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [segmentID=454029151015046147] [scope=Historical] [info="Pks range[08c9d7f4-cc77-4948-bde9-5344f1d15466-08c9d7f4-cc77-4948-bde9-5344f1d15466], len: 1, timestamp range: [454508676251910146-454508676251910146]"]

删除失败（返回成功）的query node日志显示：

[2024/12/10 06:43:35.359 +00:00] [INFO] [pipeline/delete_node.go:57] ["pipeline fetch delete msg"] [collectionID=453784516644544182] [partitionID=-1] [insertRowNum=1] [timestampMin=454508679056850945] [timestampMax=454508679056850945]
[2024/12/10 06:43:35.359 +00:00] [DEBUG] [delegator/delegator_data.go:183] ["start to process delete"] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [replicaID=454028779275157507] [ts=454508679056850946]
[2024/12/10 06:43:35.561 +00:00] [DEBUG] [delegator/delegator.go:492] ["query segments..."] [traceID=b97eb3b8b88d5edb1e299912fd4fb28a] [collectionID=453784516644544182] [channel=by-dev-rootcoord-dml_5_453784516644544182v0] [replicaID=454028779275157507] [sealedNum=1] [growingNum=1]

注意到删除失败日志中delegator没有后续删除操作，请问可能是什么原因，如何解决？

[yhmo]

search/query 对数据的可见性由consistency level决定，https://milvus.io/docs/consistency.md#Consistency-Level
delete操作是异步由querynode/datanode执行，客户端调用delete时，proxy把delete消息写入pulsar即返回，这时querynode/datanode还暂未接收到delete消息。如果此时立即查询，是有可能看到该条数据，因为真正的delete操作尚未完成。

调用search/query时设置consistency_level="Strong"即可，它是对数据的一致性做强保证，不过“Strong”会增加200ms~400ms的延时。默认一般是Bounded level，有一个时间窗口容忍数据的不一致性。

[elpsycongroo]

刚才看了下，对于多次删除不掉的数据可以稳定复现，执行删除操作后不报错但无后续删除动作

[yhmo]

v2.4.5确实有一些delete相关的bug， 在后续版本被修复。
比如：
v2.4.8 Rectified a problem where delete messages failed to be forwarded due to incorrect data scope handling. (#35313)
v2.4.6 Corrected a problem where queries could return deleted records (#34502).

所以建议升级到2.4的最新版本以避免遇到此类问题。

[elpsycongroo]

感谢，请问2.4.17相较于2.4.5有没有什么breaking change 需要修改一些配置的

[yhmo]

小版本的迭代主要是一些bug fix和enhancement，不会有不兼容的问题。