Block and Detail: Scaffolding Sketch-to-Image Generation

We introduce a novel sketch-to-image tool that aligns with the iterative refinement process of artists. Our tool lets users sketch blocking strokes to coarsely represent the placement and form of objects and detail strokes to refine their shape and silhouettes. We develop a two-pass algorithm for generating high-fidelity images from such sketches at any point in the iterative process. In the first pass we use a ControlNet to generate an image that strictly follows all the strokes (blocking and detail) and in the second pass we add variation by renoising regions surrounding blocking strokes. We also present a dataset generation scheme that, when used to train a ControlNet architecture, allows regions that do not contain strokes to be interpreted as not-yet-specified regions rather than empty space. We show that this partial-sketch-aware ControlNet can generate coherent elements from partial sketches that only contain a small number of strokes. The high-fidelity images produced by our approach serve as scaffolds that can help the user adjust the shape and proportions of objects or add additional elements to the composition. We demonstrate the effectiveness of our approach with a variety of examples and evaluative comparisons.

我们引入了一种新颖的草图到图像工具，它与艺术家的迭代细化过程相一致。我们的工具让用户绘制块状笔触来粗略地表示物体的位置和形态，并绘制细节笔触来细化它们的形状和轮廓。我们开发了一种两步算法，用于在迭代过程中的任何阶段从这些草图生成高保真图像。在第一步中，我们使用一个ControlNet来生成严格遵循所有笔触（块状和细节）的图像，在第二步中，我们通过对块状笔触周围的区域重新添加噪声来增加变化。我们还提出了一种数据集生成方案，当用于训练ControlNet架构时，它允许将不包含笔触的区域解释为尚未指定的区域，而不是空白空间。我们表明，这种具有部分草图感知能力的ControlNet能够从仅包含少量笔触的部分草图生成连贯的元素。我们的方法生成的高保真图像可作为支架，帮助用户调整物体的形状和比例，或者向构图中添加额外的元素。我们通过各种示例和评估比较证明了我们方法的有效性。