Paper Info
Title
IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning
Abstract
Conditional image generation is an active research topic including text2image and image translation. Recently image manipulation with linguistic instruction brings new challenges of multimodal conditional generation. However, traditional conditional image generation models mainly focus on generating high-quality and visually realistic images, and lack resolving the partial consistency between image and instruction. To address this issue, we propose an Increment Reasoning Generative Adversarial Network (IR-GAN), which aims to reason the consistency between visual increment in images and semantic increment in instructions. First, we introduce the word-level and instruction-level instruction encoders to learn user's intention from history-correlated instructions as semantic increment. Second, we embed the representation of semantic increment into that of source image for generating target image, where source image plays the role of referring auxiliary. Finally, we propose a reasoning discriminator to measure the consistency between visual increment and semantic increment, which purifies user's intention and guarantees the good logic of generated target image. Extensive experiments and visualization conducted on two datasets show the effectiveness of IR-GAN.
Year
DOI
Venue
2020
10.1145/3394171.3413777
MM '20: The 28th ACM International Conference on Multimedia Seattle WA USA October, 2020
DocType
ISSN
ISBN
Conference
Proceedings of the 28th ACM International Conference on Multimedia,2020
978-1-4503-7988-5
Citations 
PageRank 
References 
0
0.34
12
Authors
7
Name
Order
Citations
PageRank
Zhenhuan Liu102.03
Jincan Deng251.76
Liang Li334224.75
Shaofei Cai411.03
Qianqian Xu516022.98
Shuhui Wang659651.45
Qingming Huang73919267.71