FEAT: Supervisor supports restarts

[paradin]

Fixes #2402

Solving the issue where all workers need to be restarted after the Supervisor is restarted.

Plan:

1. Workers detect the disconnection from the Supervisor;
2. When workers reconnect to the Supervisor, if there are models running, synchronize the model replica information to the Supervisor;
3. When the Supervisor receives the model replica information synchronized by the workers, it reconstructs the data related to the model replicas.

[qinxuye]

Please fix lint, your can at the project level, run pre-commit install to do commit check.( ensure pre-commit library installed).

[qinxuye]

Really appreciate your work! I think in order to ensure this work, we may need to add integrated tests.

https://github.com/xorbitsai/inference/tree/main/xinference/core/tests

We can put the test here, maybe we can start a supervisor and worker, then kill the supervisor, restart it, to see if the model replica info successfully synced.

[qinxuye]

Lint failed, please fix it.

[qinxuye]

Tests seem all failed.

[qinxuye]

I fixed the tests, the test can run now(only run in GPU CI), but it seemed the sync_worker did not work, can you try to make the test work?

We are gonna release a new version today, hopefully we can catch up with it.

[qinxuye]

Now the CI failed at the model cannot be found after supervisor restarted.

[paradin]

I fixed the tests, the test can run now(only run in GPU CI), but it seemed the sync_worker did not work, can you try to make the test work?

We are gonna release a new version today, hopefully we can catch up with it.

Maybe the supervisor is restarted too fast without sleep.

[qinxuye]

Looks like it cannot pass still.