Distributed and Federated Training

1. What is the global batch for 8 workers with batch 32 and no accumulation?
2. What is an all-reduce of gradients for?
3. Why does the scheduler depend on the number of workers?
4. What key difference separates DDP and federated learning?
5. [E] What control validates a distributed run?

Answers. 1. 256. 2. To give the same aggregated gradient to the replicas. 3. The number of updates or the global batch may change. 4. Federated clients are separate, partial and often non-IID. 5. A single-worker comparison on equivalent data and global batch.

Activity — Single-worker equivalence

Compute the global batch for 8 workers, local batch 32 and accumulation 2. Then write a test comparing a single-worker and distributed update on the same data. Success: explained numerical gap, seeds and sampling order recorded.