Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild

Introduction

3D object detection and pose estimation methods have become popular in recent years since they can handle ambiguities in 2D images and also provide a richer description for objects compared to 2D object detectors. However, most of the datasets for 3D recognition are limited to a small amount of images per category or are captured in controlled environments. In this paper, we contribute PASCAL3D+ dataset, which is a novel and challenging dataset for 3D object detection and pose estimation. PASCAL3D+ augments 12 rigid categories of the PASCAL VOC 2012 [1] with 3D annotations. Furthermore, more images are added for each category from ImageNet [2]. PASCAL3D+ images exhibit much more variability compared to the existing 3D datasets, and on average there are more than 3,000 object instances per category. We believe this dataset will provide a rich testbed to study 3D detection and pose estimation and will help to significantly push forward research in this area. We provide the results of variations of DPM [3] on our new dataset for object detection and viewpoint estimation in different scenarios, which can be used as baselines for the community.

Publication

• Yu Xiang, Roozbeh Mottaghi and Silvio Savarese. Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2014. bibtex, pdf

PASCAL3D+

• release1.1 ~ 7.5GB (PASCAL VOC 2012 and ImageNet images and annotations, 3D CAD models, annotation tool, VDPM code, and segmentation code)
• release1.0 ~ 1GB (PASCAL VOC 2012 train and validation images and annotations, 3D CAD models, and annotation tool)

Note on 3D Object Reconstruction

• When PASCAL3D+ is used for benchmarking 3D object reconstruction, we do NOT suggest using the 3D CAD models in PASCAL3D+ for training, since the same set of 3D CAD models is used to annotate the test set. Using the 3D CAD models in both training and testing for 3D reconstruction will be biased. Please see more detailed discussion in the Appendix of [9].

Annotation Process

CAD alignments

CAD Models

Object Detection Evaluation

• We use Average Precision (AP) as the metric to evalute object detection, where the standard 50% overlap criteria of PASCAL VOC [1] is applied.
• VDPM: modified version of DPM, where mixture components correspond to discretized viewpoints. VDPM is trained on the PASCAL VOC 2012 train set, and tested on the PASCAL VOC 2012 val set.
• DPM-VOC+VP [4] is trained on the PASCAL VOC 2012 train set, and tested on the PASCAL VOC 2012 val set.
• RCNN [5] is trained on the PASCAL VOC 2012 train set by fine-tuning a pre-trained CNN on ImageNet images, and tested on the PASCAL VOC 2012 val set.

Object Detection and Pose Estimation Evaluation

• We propose a new metric called Average Viewpoint Precision (AVP) to evalute object detection and pose estimation joinly similar to AP in object detection. In computing the precision of AVP, an output from the detector is considered to be correct if and only if the bounding box overlap is larger than 50% AND the viewpoint is correct (i.e., the two viewpoint labels are the same in discrete viewpoint space or the distance between the two viewpoints is smaller than some threshold in continuous viewpoint space). The recall is the same for AP and AVP. As a result, AP is always an upper bound of AVP.
• VDPM: modified version of DPM, where mixture components correspond to discretized viewpoints. VDPM is trained on the PASCAL VOC 2012 train set, and tested on the PASCAL VOC 2012 val set.
• DPM-VOC+VP [4] is trained on the PASCAL VOC 2012 train set, and tested on the PASCAL VOC 2012 val set.
• Viewpoints & Keypoints [6] is trained on the PASCAL VOC 2012 train set and the ImageNet images in PASCAL3D+, and tested on the PASCAL VOC 2012 val set with object detections from RCNN [5].

Add Your Results

• Send your detection and pose estimation results to < yuxiang at cs dot stanford dot edu >.
• Ideal format: for each test image, a set of detected bounding boxes with detection scores and viewpoints.

Related Datasets

• 3D Object Dataset: a benchmark for object detection and pose estimation (10 categories with 10 object instances for each category).
• EPFL Car Dataset: a multi-view car dataset for pose estimation (20 car instances).
• KITTI Detection Dataset: a street scene dataset for object detection and pose estimation (3 categories: car, pedestrian and cyclist).
• PASCAL VOC Detection Dataset: a benchmark for 2D object detection (20 categories).
• SUN3D: a database of big spaces reconstructed using SfM and object labels.
• LabelMe3D: a database of 3D scenes from user annotations.
• NYC3DCars: a database of 3D vehicles in geographic context.

Acknowledgements

• We acknowledge the support of ONR grant N00014-13-1-0761 and NSF CAREER grant #1054127. We thank Taewon Kim, Yawei Wang and Jino Kim for their valuable help in building this benchmark. We thank Bojan Pepik for his help in conducting the experiments with DPM-VOC+VP.

References

• M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
• J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
• P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. PAMI, 2010.
• B. Pepik, M. Stark, P. Gehler, and B. Schiele. Teaching 3d geometry to deformable part models. In CVPR, 2012.
• R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
• S. Tulsiani and J. Malik. Viewpoints and keypoints. In CVPR, 2015.
• F. Massa, R. Marlet and M. Aubry. Crafting a multi-task CNN for viewpoint estimation. In BMVC, 2016.
• P. Poirson, P. Ammirato, C.Y. Fu, W. Liu, J. Kosecka and A. Berg. Fast Single Shot Detection and Pose Estimation. In 3DV, 2016.
• S. Tulsiani, T. Zhou, A.A. Efros and J. Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In CVPR, 2017.

Contact : yuxiang at cs dot stanford dot edu

Last update : 6/29/2017