Commit 49362b1
Tutorial combining DDP with Pipeline Parallelism to Train Transformer models (#1347)
* Tutorial combining DDP with Pipeline Parallelism to Train Transformer models.
Summary: Tutorial which places a pipe on GPUs 0 and 1 and another Pipe
on GPUs 2 and 3. Both pipe replicas are replicated via DDP. One process
drives GPUs 0 and 1 and another drives GPUs 2 and 3.
* Polish out some of the docs.
* Add thumbnail and address some comments.
Co-authored-by: pritam <pritam.damania@fb.com>1 parent 07b6340 commit 49362b1
File tree
3 files changed
+472
-0
lines changed- _static/img/thumbnails/cropped
- advanced_source
3 files changed
+472
-0
lines changed
0 commit comments