The Communication Complexity of Optimization

We consider the communication complexity of a number of distributed optimization problems. We start with the problem of solving a linear system. Suppose there is a coordinator together with $s$ servers $P_1, \ldots, P_s$, the $i$-th of which holds a subset $A^{(i)} x = b^{(i)}$ of $n_i$ constraints of a linear system in $d$ variables, and the coordinator would like to output $x \in \mathbb{R}^d$ for which $A^{(i)} x = b^{(i)}$ for $i = 1, \ldots, s$. We assume each coefficient of each constraint is specified using $L$ bits. We first resolve the randomized and deterministic communication complexity in the point-to-point model of communication, showing it is $\tilde{\Theta}(d^2L + sd)$ and $\tilde{\Theta}(sd^2L)$, respectively. We obtain similar results for the blackboard model. When there is no solution to the linear system, a natural alternative is to find the solution minimizing the $\ell_p$ loss. While this problem has been studied, we give improved upper or lower bounds for every value of $p \ge 1$. One takeaway message is that sampling and sketching techniques, which are commonly used in earlier work on distributed optimization, are neither optimal in the dependence on $d$ nor on the dependence on the approximation $\epsilon$, thus motivating new techniques from optimization to solve these problems. Towards this end, we consider the communication complexity of optimization tasks which generalize linear systems. For linear programming, we first resolve the communication complexity when $d$ is constant, showing it is $\tilde{\Theta}(sL)$ in the point-to-point model. For general $d$ and in the point-to-point model, we show an $\tilde{O}(sd^3 L)$ upper bound and an $\tilde{\Omega}(d^2 L + sd)$ lower bound. We also show if one perturbs the coefficients randomly by numbers as small as $2^{-\Theta(L)}$, then the upper bound is $\tilde{O}(sd^2 L) + \textrm{poly}(dL)$.

我们考虑许多分布式优化问题的通信复杂性。我们从解决线性系统的问题开始。假设有一个协调器与$ s $ servers $ p_1，\ ldots，p_s $，其中$ i $ - the $ i $ th保持子集$ a^{（i）} x = b^{（i）} $ $ n_i $ $ d $变量中的线性系统的约束，并且协调器希望输出$ x \ in \ mathbb {r}^d $ $ a^{（i）} x = b^{（i）} $ for $ i = 1，\ ldots，s $。我们假设使用$ L $位指定每个约束的每个系数。我们首先在通信的点对点模型中解决了随机和确定性的通信复杂性，表明它是$ \ tilde {\ theta}（d^2l + sd）$和$ \ tilde {\ tilde {\ theta}（sd^2l ）$，分别。我们为黑板模型获得了类似的结果。当没有解决线性系统的解决方案时，一种自然的选择是找到最小化$ \ ell_p $损失的解决方案。尽管已经研究了此问题，但我们为$ p \ ge 1 $的每个值提供了改进的上限或下限。一个要点消息是，在较早的分布式优化方面通常使用的抽样和草图技术在$ d $的依赖性方面既不是最佳的，也不是对近似值$ \ epsilon $的依赖，从而激发了从优化到优化的新技术解决这些问题。为此，我们考虑了概括线性系统的优化任务的通信复杂性。对于线性编程，我们首先在$ d $是恒定的情况下解决通信复杂性，表明它是点对点模型中的$ \ tilde {\ theta}（sl）$。对于一般$ d $，在点对点模型中，我们显示$ \ tilde {o}（sd^3 l）$上限和$ \ tilde {\ omega}（d^2 l + sd） $下限。我们还表明，是否会随机按数字随机随机，$ 2^{ - \ theta（l）} $，那么上限为$ \ tilde {o}（sd^2 l） + \ textrm {poly}（poly}（poly}）（ DL）$。