[core] TSAN failure for GcsHealthCheckManagerTest

[dentiny]

What happened + What you expected to happen

[2024-12-27T20:44:08Z] ==================
--
  | [2024-12-27T20:44:08Z] WARNING: ThreadSanitizer: data race (pid=1309)
  | [2024-12-27T20:44:08Z]   Write of size 8 at 0x7b2400001b00 by main thread:
  | [2024-12-27T20:44:08Z]     #0 free /tmp/llvm/utils/release/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:707:3 (gcs_health_check_manager_test+0x9c958)
  | [2024-12-27T20:44:08Z]     #1 std::__1::default_delete<ray::gcs::GcsHealthCheckManager>::operator()(ray::gcs::GcsHealthCheckManager*) const /opt/llvm/bin/../include/c++/v1/memory:1423:5 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #2 std::__1::unique_ptr<ray::gcs::GcsHealthCheckManager, std::__1::default_delete<ray::gcs::GcsHealthCheckManager> >::reset(ray::gcs::GcsHealthCheckManager*) /opt/llvm/bin/../include/c++/v1/memory:1684:7 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #3 std::__1::unique_ptr<ray::gcs::GcsHealthCheckManager, std::__1::default_delete<ray::gcs::GcsHealthCheckManager> >::~unique_ptr() /opt/llvm/bin/../include/c++/v1/memory:1638:19 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #4 GcsHealthCheckManagerTest::~GcsHealthCheckManagerTest() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:53:7 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #5 GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test::~GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:165:1 (gcs_health_check_manager_test+0x110429)
  | [2024-12-27T20:44:08Z]     #6 testing::Test::DeleteSelf_() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:336:24 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd67a2)
  | [2024-12-27T20:44:08Z]     #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #8 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #9 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2842:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd6442)
  | [2024-12-27T20:44:08Z]     #10 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd79e4)
  | [2024-12-27T20:44:08Z]     #11 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5920:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xecc44)
  | [2024-12-27T20:44:08Z]     #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #14 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5484:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xebe4c)
  | [2024-12-27T20:44:08Z]     #15 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2317:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]     #16 main /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z]   Previous read of size 8 at 0x7b2400001b00 by thread T30:
  | [2024-12-27T20:44:08Z]     #0 ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1::operator()(grpc::Status) const /proc/self/cwd/src/ray/gcs/gcs_server/gcs_health_check_manager.cc:139:19 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #1 decltype(std::__1::forward<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&>(fp)(std::__1::forward<grpc::Status>(fp0))) std::__1::__invoke<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status>(ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status&&) /opt/llvm/bin/../include/c++/v1/type_traits:3694:1 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #2 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status>(ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status&&) /opt/llvm/bin/../include/c++/v1/__functional_base:348:9 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #3 std::__1::__function::__alloc_func<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1, std::__1::allocator<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1>, void (grpc::Status)>::operator()(grpc::Status&&) /opt/llvm/bin/../include/c++/v1/functional:1558:16 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #4 std::__1::__function::__func<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1, std::__1::allocator<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1>, void (grpc::Status)>::operator()(grpc::Status&&) /opt/llvm/bin/../include/c++/v1/functional:1732:12 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #5 std::__1::__function::__value_func<void (grpc::Status)>::operator()(grpc::Status&&) const /opt/llvm/bin/../include/c++/v1/functional:1885:16 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #6 std::__1::function<void (grpc::Status)>::operator()(grpc::Status) const /opt/llvm/bin/../include/c++/v1/functional:2560:12 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #7 void grpc::internal::CatchingCallback<std::__1::function<void (grpc::Status)>, grpc::Status>(std::__1::function<void (grpc::Status)>&&, grpc::Status&&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:43:5 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #8 grpc::internal::CallbackWithStatusTag::Run(bool) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:128:5 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a3cf)
  | [2024-12-27T20:44:08Z]     #9 grpc::internal::CallbackWithStatusTag::StaticRun(grpc_completion_queue_functor*, int) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:112:46 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a0d4)
  | [2024-12-27T20:44:08Z]     #10 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)::operator()(void*) const /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:94:17 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd13df)
  | [2024-12-27T20:44:08Z]     #11 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)::__invoke(void*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:62:13 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd13df)
  | [2024-12-27T20:44:08Z]     #12 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:145:11 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2c46a)
  | [2024-12-27T20:44:08Z]     #13 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:115:9 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2c46a)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z]   Thread T30 'nexting_thread' (tid=1347, running) created by main thread at:
  | [2024-12-27T20:44:08Z]     #0 pthread_create /tmp/llvm/utils/release/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (gcs_health_check_manager_test+0x9db2b)
  | [2024-12-27T20:44:08Z]     #1 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:113:30 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2bf36)
  | [2024-12-27T20:44:08Z]     #2 grpc_core::Thread::Thread(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:199:15 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2bf36)
  | [2024-12-27T20:44:08Z]     #3 void std::__1::allocator<grpc_core::Thread>::construct<grpc_core::Thread, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(grpc_core::Thread*, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/memory:911:28 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #4 void std::__1::allocator_traits<std::__1::allocator<grpc_core::Thread> >::construct<grpc_core::Thread, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&, void>(std::__1::allocator<grpc_core::Thread>&, grpc_core::Thread*, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:288:13 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #5 void std::__1::vector<grpc_core::Thread, std::__1::allocator<grpc_core::Thread> >::__emplace_back_slow_path<char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/vector:1667:5 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #6 grpc_core::Thread& std::__1::vector<grpc_core::Thread, std::__1::allocator<grpc_core::Thread> >::emplace_back<char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/vector:1687:9 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #7 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:60:26 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #8 grpc::CompletionQueue::CallbackAlternativeCQ() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:194:36 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #9 grpc::Server::CallbackCQ() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1386:19 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xf1112)
  | [2024-12-27T20:44:08Z]     #10 grpc::Server::RegisterService(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*, grpc::Service*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1074:35 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xee3f5)
  | [2024-12-27T20:44:08Z]     #11 grpc::Server::Start(grpc::ServerCompletionQueue**, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1172:5 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xef5e7)
  | [2024-12-27T20:44:08Z]     #12 grpc::ServerBuilder::BuildAndStart() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_builder.cc:446:11 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xe71f7)
  | [2024-12-27T20:44:08Z]     #13 ray::rpc::GrpcServer::Run() /proc/self/cwd/src/ray/rpc/grpc_server.cc:122:21 (liblibgrpc_Ucommon_Ulib.so+0x7495a)
  | [2024-12-27T20:44:08Z]     #14 GcsHealthCheckManagerTest::AddServer(bool) /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:90:13 (gcs_health_check_manager_test+0x10da06)
  | [2024-12-27T20:44:08Z]     #15 GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test::TestBody() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:166:18 (gcs_health_check_manager_test+0x10dfe6)
  | [2024-12-27T20:44:08Z]     #16 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #17 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #18 testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2687:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4901)
  | [2024-12-27T20:44:08Z]     #19 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2836:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd63c8)
  | [2024-12-27T20:44:08Z]     #20 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd79e4)
  | [2024-12-27T20:44:08Z]     #21 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5920:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xecc44)
  | [2024-12-27T20:44:08Z]     #22 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #23 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #24 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5484:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xebe4c)
  | [2024-12-27T20:44:08Z]     #25 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2317:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]     #26 main /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z] SUMMARY: ThreadSanitizer: data race /opt/llvm/bin/../include/c++/v1/memory:1423:5 in std::__1::default_delete<ray::gcs::GcsHealthCheckManager>::operator()(ray::gcs::GcsHealthCheckManager*) const
  | [2024-12-27T20:44:08Z] ==================

Versions / Dependencies

master

Reproduction script

N/A

Issue Severity

Low: It annoys or frustrates me.