How to run fedscale on multiple GPUs？

[Geeksun2018]

I have four GPUs. According to the tutorial, I set up the worker_ IPS is [4,4,4,4], but the training speed is not even faster than that of a single GPU (worker_ips = [4]), so I doubt if there is a problem with my settings.

[AmberLJC]

Can I see you config file? It might be related to your num_participants, dataset etc.

[Geeksun2018]

conf.zip
I modified the overcommitment,use_cuda,local_steps,num_participants and worker_ips

[AmberLJC]

Thanks for the feedback and let me try to reproduce first.

[AmberLJC]

Hi I cannot reproduce your results. In my experiments, [4,4,4,4](1st figure) is faster than [4](2nd figure) roughly 3-4 times following the same setting.
There also might be problem with our GPU or network. Hope these help.

[fanlai0990]

Thanks for the prompt response! There are two points that may help:

• Please check the leftmost timestamp for the evaluation speed. Note that "Wall clock" is the FL runtime simulated using realistic FL trace;
• Please double check whether you have fedscale driver stop $JOB_NAME before submitting a new job. Otherwise, concurrent jobs lead to resource contention, thus slower evaluation;
• Please consider to use overcommitment > 1 (in real FL it is often set to 1.3) when we are activating device_trace in configs. This is because FedScale will integrate the client availability trace after specifying the device_trace. So the number of completion we will receive will be slightly smaller than the number of participants we selected, but overcommitment > 1 will ensure we are selecting more clients. e.g., check the "Succeed participants:" vs. "Planned participants" in these two figures. If you don't care about realistic client availability, just comment out device_trace.

Hope this can help. Thank!

[Geeksun2018]

Thank you very much. I will check my code carefully again!