ACT is working, but not Diffusion

[Kacchan16]

Hello Team,

your work is so good, I am currently working on creating some nice policies with Lerobot repo, architecture and software. I tried ACT on my robot, it is working fine, able to execute the tasks what it learnt in the evaluation.
I tried training Diffusion policy, multiple times with different params and also the default params, what you provided in the repo. I tried PushT in colab, its working but not in robot. Can you please explain why its not working, or should I change other things??
I forgot to mention, I used 3 cameras for data collection and training for Diffusion
Thank you

EDIT (aliberts): format

[Cadene]

Thanks for reaching out @Kacchan16

Could you provide more info? What are the behaviors that you observe with diffusion?

What configs did you try?

To be honest, we never tried to make it work on real task because of the long inference time + our priorities on hardware+software dev. We were hoping that people from the community would figure it out ^^

[Kacchan16]

config.txt
Thanks for quick reply, and sorry I forgot to mention the task, I am trying picking and placing the object first.

this is the config, I tried changing the resnet18 to resnet34, and offline_steps:200000 to 100000.
I reduced batch_size from 64 to 16, 8 bcoz of time and storage. and in starting I commented observation.state and action now I am training with the stats for my dataset.

Then Lerobot Diffusion wont work on real robot ?? ;(

[Kacchan16]

Hi all and @Cadene ,
I just wanted to ask one more question continous to previous one, Can you please just tell me this, Diffusion is not gonna work on real robot, what about VQ-BET and TDMPC ?

I thought of trying those policies on real robots. If you just tell me its gonna work or not, it will saves more time, rather than training with multiple parameters to make it work.

Thank you :)

[Cadene]

Diffusion is expected to work on real robot ; but it requires some tuning imo.
Same for VQ-BET.

TDMPC is RL. It requires some rewards. We are working on making RL work well on real robots ;)