A Lagrangian Perspective on Latent Variable Generative Models

A large number of objectives have been proposed to train latent variable generative models. We show that many of them are Lagrangian dual functions of the same primal optimization problem. The primal problem optimizes the mutual information between latent and visible variables, subject to the constraints of accurately modeling the data distribution and performing correct amortized inference. By choosing to maximize or minimize mutual information, and choosing different Lagrange multipliers, we obtain different objectives including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial autoencoders, AVB, AS-VAE and InfoVAE. Based on this observation, we provide an exhaustive characterization of the statistical and computational trade-offs made by all the training objectives in this class of Lagrangian duals. Next, we propose a dual optimization method where we optimize model parameters as well as the Lagrange multipliers. This method achieves Pareto optimal solutions in terms of optimizing information and satisfying the constraints.

已经提出了大量目标来训练潜在变量生成模型。我们表明，其中许多目标是同一个原始优化问题的拉格朗日对偶函数。原始问题在准确建模数据分布以及进行正确的摊销推断的约束条件下，优化潜在变量和可见变量之间的互信息。通过选择最大化或最小化互信息，并选择不同的拉格朗日乘子，我们得到了不同的目标，包括InfoGAN、ALI/BiGAN、ALICE、CycleGAN、β -VAE、对抗自编码器、AVB、AS -VAE和InfoVAE。基于这一观察，我们对这类拉格朗日对偶中的所有训练目标所做的统计和计算权衡进行了详尽的描述。接下来，我们提出一种对偶优化方法，在该方法中我们同时优化模型参数以及拉格朗日乘子。这种方法在优化信息和满足约束方面实现了帕累托最优解。