Cloud2Sketch: Augmenting Clouds with Imaginary Sketches
🖌️

Cloud2Sketch: Augmenting Clouds with Imaginary Sketches

📖
In Proceedings of ACM International Conference on Multimedia. (ACM MM’22)

†Co-corresponding authors.

Abstract

Have you ever looked up at the sky and imagined what the clouds look like? In this work, we present an interesting task that augments clouds in the sky with imagined sketches. Different from generic image-to-sketch translation tasks, unique challenges are introduced: real-world clouds have different levels of similarity to something; sketch generation without sketch retrieval could lead to something unrecognizable; a retrieved sketch from some dataset cannot be directly used because of the mismatch of the shape; an optimal sketch imagination is subjective. We propose Cloud2Sketch, a novel self-supervised pipeline to tackle the aforementioned challenges. First, we pre-process cloud images with a cloud detector and a thresholding algorithm to obtain cloud contours. Then, cloud contours are passed through a retrieval module to retrieve sketches with similar geometrical shapes. Finally, we adopt a novel sketch translation model with built-in free-form deformation for aligning the sketches to cloud contours. To facilitate training, an icon-based sketch collection named Sketchy Zoo is proposed. Extensive experiments validate the effectiveness of our method both qualitatively and quantitatively.

Method Review

Our proposed pipeline for augmenting clouds with imaginary sketches. We propose to decompose the task into three stages, 1) cloud contour detection to obtain cloud contours as shape features, 2) sketch retrieval to search for geometrically similar sketch candidates, and 3) sketch alignment to smoothly align retrieved sketches and produce drawings on cloud.
Our proposed pipeline for augmenting clouds with imaginary sketches. We propose to decompose the task into three stages, 1) cloud contour detection to obtain cloud contours as shape features, 2) sketch retrieval to search for geometrically similar sketch candidates, and 3) sketch alignment to smoothly align retrieved sketches and produce drawings on cloud.

Downloads

SketchyZoo

Citation

@inproceedings{10.1145/3503161.3547810, author = {Wan, Zhaoyi and Xu, Dejia and Wang, Zhangyang and Wang, Jian and Luo, Jiebo}, title = {Cloud2Sketch: Augmenting Clouds with Imaginary Sketches}, year = {2022}, isbn = {9781450392037}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3503161.3547810}, doi = {10.1145/3503161.3547810}, booktitle = {Proceedings of the 30th ACM International Conference on Multimedia}, pages = {2441–2451}, numpages = {11}, keywords = {cloud augmentation, sketch synthesis, shape alignment, image-to-sketch generation}, location = {Lisboa, Portugal}, series = {MM '22} }

Acknowledgements