How to detect and track the foot using ARKIt and vision framework?

Question

I wants to virtually add foot ball and detect and tracking the foot so that we can simulate the kick to the ball.

Can any one please suggest the way achieve it in iOS?

Answer 1

I think you'll have to have your own CoreML model that identify where a foot is in an image, and pass it the frames captured by the camera and identify where there is a foot. ARKit doesn't do that and there's no foot recognition built in iOS 11

Answer 2

There are several ready-made solutions for detecting parts of your body (including legs) provided that you are standing at full height in front of the camera (Apple ARKit can do this out of the box in the latest versions).

However, if you are looking for a solution to move your own leg - you are faced with a research and development phase.

Because the angle at which the user holds their device is not typical and the picture is quite limited in that only part of the leg and foot are visible.

You will have to:

1. Identify the feet with all control points (not only to determine whether it is the left or right foot in the frame but also to understand how it is positioned)
2. Draw a 3D model of the ball in the frame, and track the movement of the control points of the foot until they touch the model with the ball. (In this case, you will need to take into account the vector of movement of the leg in the frame in order to understand in which direction you should move your 3D model with the ball. In addition, take into account acceleration in order to understand at what speed you will need to move the ball after the moment of contact with him).

The problem is that using native ARKit (the iOS way) for example, you can render the 3D model and even control it by control points, so it doesn't really matter. But the main problem with step 1 is finding the legs with all the control points at an atypical angle when you are scanning your own legs and not the entire person in front of you.

So you may need an ML engineer (possibly in Pyhton) to create the ML model first. Then use a native AR platform (such as ARKit for iOS) or Unity (if not native) to obtain and render the model relative to the detected legs.

https://www.youtube.com/watch?app=desktop&v=kcsCf4SlFLs