Cloud2Sketch: Augmenting Clouds with Imaginary Sketches

†Co-corresponding authors.

Have you ever looked up at the sky and imagined what the clouds look like? In this work, we present an interesting task that augments clouds in the sky with imagined sketches. Different from generic image-to-sketch translation tasks, unique challenges are introduced: real-world clouds have different levels of similarity to something; sketch generation without sketch retrieval could lead to something unrecognizable; a retrieved sketch from some dataset cannot be directly used because of the mismatch of the shape; an optimal sketch imagination is subjective. We propose Cloud2Sketch, a novel self-supervised pipeline to tackle the aforementioned challenges. First, we pre-process cloud images with a cloud detector and a thresholding algorithm to obtain cloud contours. Then, cloud contours are passed through a retrieval module to retrieve sketches with similar geometrical shapes. Finally, we adopt a novel sketch translation model with built-in free-form deformation for aligning the sketches to cloud contours. To facilitate training, an icon-based sketch collection named Sketchy Zoo is proposed. Extensive experiments validate the effectiveness of our method both qualitatively and quantitatively.

@inproceedings{10.1145/3503161.3547810, author = {Wan, Zhaoyi and Xu, Dejia and Wang, Zhangyang and Wang, Jian and Luo, Jiebo}, title = {Cloud2Sketch: Augmenting Clouds with Imaginary Sketches}, year = {2022}, isbn = {9781450392037}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3503161.3547810}, doi = {10.1145/3503161.3547810}, booktitle = {Proceedings of the 30th ACM International Conference on Multimedia}, pages = {2441–2451}, numpages = {11}, keywords = {cloud augmentation, sketch synthesis, shape alignment, image-to-sketch generation}, location = {Lisboa, Portugal}, series = {MM '22} }