Can we use android ML Kit Image Labeling to detect a product

Question

I am having a requirement from client that we need to detect our product box in cameraView. For that I am trying to use firebase ML Kit object detection. For now it detects general objects like door, sofa etc. I want to be able to just detect my product. Is there anyway to do that? Below is my current implementation

class DetectActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_detect)
    cameraView.setLifecycleOwner(this)
    cameraView.addFrameProcessor {
        extractDataFromFrame(it) { result ->
            tvDetectedItem.text = result
        }
    }
}

private fun extractDataFromFrame(frame: Frame, callback: (String) -> Unit) {

    val options = FirebaseVisionObjectDetectorOptions.Builder()
            .setDetectorMode(FirebaseVisionObjectDetectorOptions.STREAM_MODE)
            .enableMultipleObjects()
            .enableClassification()  // Optional
            .build()

    val objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options)

    objectDetector.processImage(getVisionImageFromFrame(frame))
            .addOnSuccessListener {
                var result = ""
                it.forEach { item ->
                    result += "${item.entityId}\n"  //TODO : Get the knowledge graph result for this entity
                    Log.e("TAG",item.classificationCategory.toString())
                }
                callback(result)
            }
            .addOnFailureListener {
                callback("Unable to detect an object")
            }
            .addOnCompleteListener {

            }

}

private fun getVisionImageFromFrame(frame : Frame) : FirebaseVisionImage{
    //ByteArray for the captured frame
    val data = frame.data

    //Metadata that gives more information on the image that is to be converted to FirebaseVisionImage
    val imageMetaData = FirebaseVisionImageMetadata.Builder()
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(FirebaseVisionImageMetadata.ROTATION_90)
            .setHeight(frame.size.height)
            .setWidth(frame.size.width)
            .build()

    val image = FirebaseVisionImage.fromByteArray(data, imageMetaData)

    return image
}

}

Also I want it to be completely offline. For that there is on device ML kit implementation. But how can I use it? Sorry for my bad english.

Answer 1

First, firebase ML Kit object detection is deprecated and no longer supported. Please migrate to the standalone ML Kit (see migration guide). Secondly, for your purpose, you can check out ML Kit's Object Detection and Tracking with custom models (developer guide). To train your own image classification models for your products, ML Kit's custom models developer guide talks about several methods. Good luck!

Answer 2

we made some changes to Firebase ML Kit to better distinguish the on-device APIs from cloud based APIs. "ML Kit"(without firebase branding) contains all the on-device APIs. Here's the migration guide from firebase mlkit to mlkit. All further improvements and new APIs will be released only with the new ML Kit.

The default model used by the object detection sdk is very simple. You can apply a custom model to the obejct detection API now with the standalone ML Kit. There are some free powerful model compatible by MLKit API there in tfhub. I found this one very powerful which you can give it a try.