The Minimax Complexity of Federated Optimization under Interpolation: The Role of Oracle Models

Abstract

We study the minimax complexity of smooth convex stochastic optimization with intermittent communication: $m$ workers run for $N$ rounds, and between consecutive communication rounds each worker makes $K$ sequential stochastic gradient queries. We focus on the interpolation regime, in which all sample losses share a common minimizer, so that the gradient noise vanishes at the solution. Let $L$ be the smoothness of the objective, $H_s\ge L$ the smoothness of the sample losses, and $R$ the distance from the initial point to a minimizer. We consider three oracle models that provide increasing information: an unbiased stochastic gradient oracle $g(x,ξ)$ satisfying $\mathbb{E}_ξ\|g(x,ξ)\|^2 \leq 2H_s\bigl(F(x)-F(x^*)\bigr)$, gradients of the sample losses at fresh samples, and repeated gradient queries to stored samples. Up to logarithmic factors, all our results are described by two rates: \[ R^2\min\left\{L,\;\frac{H_s}{KN},\; \frac{L}{N^2}+\frac{H_s}{mKN}\right\} \qquad\text{and}\qquad R^2\min\left\{L,\; \frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}, \] Under the weak growth condition, the first rate is the minimax rate, attained by the best of three simple baselines, as in the classical bounded-variance setting. With repeated access to samples, the second rate becomes the minimax rate for general convex objectives. For general convex objectives with fresh sample gradients, the minimax rate lies between the two rates. Our lower bounds apply to arbitrary randomized intermittent-communication algorithms.