Race-condition on shutdown of current-thread runtime?

[thomaseizinger]

Is your feature request related to a problem? Please describe.

In my application, I am spawning threads that use tokio's current-thread runtime. Only a single task is spawned onto this runtime via block_on. The remaining application is connected with this task via channels. When the application is shut down, there appears to be a race condition between cleaning up the runtime and polling the task that is spawned onto the runtime. In particular, my task ends up spamming "A Tokio 1.x context was found, but it is being shutdown." from polling the AsyncFd within that task.

This is the task that is running inside the runtime: https://github.com/firezone/firezone/blob/90bac881945ebe4f91f812672da2633cfe3c1079/rust/tun/src/unix.rs#L39-L107

What I don't understand is: How can the runtime shut down while the task is still being polled? I would expect that either the runtime shuts down and de-allocates the task or the runtime is still active and polls the task. Here is an example of where the thread is being spawned:

https://github.com/firezone/firezone/blob/90bac881945ebe4f91f812672da2633cfe3c1079/rust/connlib/clients/apple/src/tun.rs#L33-L53

Describe the solution you'd like

• Runtime shutdown to be atomic, i.e. a task running within block_on to never be polled if the runtime is being shut down
• A way of detecting runtime shutdown from within the task

Describe alternatives you've considered

The way I am solving this is by checking for the message of the io::Error (see firezone/firezone#7605). That isn't very clean and could break if that message every changes. It is additional API surface but perhaps a function could be exposed that matches an io::Error against it being the "shut down"-error? Or a function on Handle to detect whether the runtime is shutting down?

Additional context
Add any other context or screenshots about the feature request here.

[Darksonn]

The way you shut down the runtime is by running the destructor of the Runtime. That cannot possibly happen with the send_recv_tun function since you're inside a call to block_on on the Runtime object. Are you sure that send_recv_tun is the function where this panic happens?

Are you perhaps using tokio::runtime::Handle::block_on anywhere?

[thomaseizinger]

Are you sure that send_recv_tun is the function where this panic happens?

There is no panic! Instead, the AsyncFd returns an io::Error with that error text.

[thomaseizinger]

Are you perhaps using tokio::runtime::Handle::block_on anywhere?

Not on that runtime. The linked async fn is the only thing that runs on that runtime.

[Darksonn]

Sorry, I meant the error not panic. But I still don't see how you are shutting down the runtime without first exiting from the block_on call. Where do you trigger shutdown?

[Darksonn]

Oh. I understand now.

The problem is that the AsyncFd is using the IO driver of the runtime it was created in. It stops working when that other runtime shuts down. But the future is executing under a different runtime that did not get shut down.

You should wait with creating the AsyncFd until you're inside the new runtime.

[thomaseizinger]

Oh. I understand now.

The problem is that the AsyncFd is using the IO driver of the runtime it was created in. It stops working when that other runtime shuts down. But the future is executing under a different runtime that did not get shut down.

You should wait with creating the AsyncFd until you're inside the new runtime.

Oh! I didn't realise that is what is happening. Thanks for pointing that out :)

[thomaseizinger]

This is a bit bizarre. Perhaps there should be some kind of warning about that? If each runtime where to carry a unique identifier, this could potentially be detected?

[Darksonn]

We could probably add some logic to detect this when creating the error message, so that we could emit a better error for this case.