about device

[pwd11]

If I want to replicate your experiment, will three 4090 graphics cards suffice?

[DD-DuDa]

If you are runing 7b models, I thinks it would be ok. Just try to alter the deepspeed config.