Implement Multi-GPU Training on a single GPU

Multi-GPU Training on a single GPU System in 3 Minutes

I want to share with you a neat little trick on how I test my multi GPU training code on a single GPU.

---

Motivation

I guess the problem is obvious and you probably experienced it yourself. You want to train a Deep Learning model and you want to take advantage of multiple GPUs, a TPU or even multiple workers for some extra speed or larger batch size. But of course you cannot (let's say should not because I've seen it quite often

Tags: Deep Learning Distributed Systems Gpu Python TensorFlow

Comment