COCO object detection evaluation metrics vs Kitti evaluation

Question

For my validation dataset (own data), I evaluate using both COCO and Kitti evaluation metrics.

I expect a similar score for COCO AP (0.5) value and Kitti 2D AP (with iou=0.5). But I am not sure if they are comparable, inspite the logic behind them being the same (2D front view GT and DET boxes are checked for iou>0.5 and matched and precision/recalls are computed). 

COCO obj detection AP 0.5 = 0.59 Kitti 2D AP with iou set as 0.5 = 0.33

Is it right to expect them to be comparable?? Most papers I find use only one of them, and no one has really compared the results got from both evaluation metrics. 

Answer 1

From their website

For evaluation, we compute precision-recall curves for object detection and orientation-similarity-recall curves for joint object detection and orientation estimation. In the latter case not only the object 2D bounding box has to be located correctly, but also the orientation estimate in bird's eye view is evaluated.

(..)

We evaluate object detection performance using the PASCAL criteria and object detection and orientation estimation performance using the measure discussed in our CVPR 2012 publication

Not knowing too much about Kitti evaluation metrics, by reading this, it seems that they are not comparable to each other and might not be appropriate for your common object detection procedure.

If your data has nothing to do with the Kiti dataset and their objective, strongly recommend you to discard their metrics and use COCO metrics or PASCAL.

PASCAL is the original metric system, but COCO has been the most used in recent times since it is a more strict measure.