Abstract | ||
---|---|---|
Existing image-based activity understanding methods mainly adopt direct mapping, i.e. from image to activity concepts, which may encounter performance bottleneck since the huge gap. In light of this, we propose a new path: infer human part states first and then reason out the activities based on part-level semantics. Human Body Part States (PaSta) are fine-grained action semantic tokens, e.g. <hand, hold, something>, which can compose the activities and help us step toward human activity knowledge engine. To fully utilize the power of PaSta, we build a large-scale knowledge base PaStaNet, which contains 7M+ PaSta annotations. And two corresponding models are proposed: first, we design a model named Activity2Vec to extract PaSta features, which aim to be general representations for various activities. Second, we use a PaSta-based Reasoning method to infer activities. Promoted by PaStaNet, our method achieves significant improvements, e.g. 6.4 and 13.9 mAP on full and one-shot sets of HICO in supervised learning, and 3.2 and 4.2 mAP on V-COCO and images-based AVA in transfer learning. Code and data are available at http://hake-mvig.cn/. |
Year | DOI | Venue |
---|---|---|
2020 | 10.1109/CVPR42600.2020.00046 | 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |
Keywords | DocType | ISSN |
human activity knowledge engine,activity concepts,infer human part states,part-level semantics,Human Body Part States,large-scale knowledge base PaStaNet,Activity2Vec,PaSta features,PaSta annotations,V-COCO,images-based AVA,HICO,supervised learning,PaSta-based reasoning method | Conference | 1063-6919 |
ISBN | Citations | PageRank |
978-1-7281-7169-2 | 0 | 0.34 |
References | Authors | |
44 | 10 |
Name | Order | Citations | PageRank |
---|---|---|---|
Yonglu Li | 1 | 22 | 7.05 |
Liang Xu | 2 | 19 | 2.26 |
Xinpeng Liu | 3 | 5 | 1.75 |
Xijie Huang | 4 | 19 | 2.26 |
Xu Yue | 5 | 0 | 0.34 |
Wang Shiyi | 6 | 1 | 0.69 |
Haoshu Fang | 7 | 57 | 6.86 |
Ze Ma | 8 | 13 | 0.84 |
Chen Mingyang | 9 | 0 | 0.34 |
Cewu Lu | 10 | 993 | 62.08 |