Name
Title | ||
|---|---|---|
SketchParse: Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks. |
Abstract | ||
|---|---|---|
The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SKETCHPARSE, the first deep-network architecture for fully automatic parsing of freehand object sketches. SKETCHPARSE is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SKETCHPARSE's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SKETCHPARSE's output can be used to generate caption-style descriptions for hand-drawn sketches.
|
Year | DOI | Venue |
|---|---|---|
2017 | 10.1145/3123266.3123270 | MM '17: ACM Multimedia Conference
Mountain View
California
USA
October, 2017 |
Keywords | DocType | Volume |
object segmentation, sketch, transfer learning, deep learning, multi-task learning | Journal | abs/1709.01295 |
ISBN | Citations | PageRank |
978-1-4503-4906-2 | 1 | 0.35 |
References | Authors | |
41 | 5 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Ravi Kiran Sarvadevabhatla | 1 | 7 | 8.41 |
| Isht Dwivedi | 2 | 1 | 0.35 |
| Abhijat Biswas | 3 | 1 | 0.35 |
| Sahil Manocha | 4 | 1 | 0.35 |
| R. Venkatesh Babu | 5 | 1046 | 84.83 |