Webots robot doesn't speak while trying to use it with an external Kinect

Question

I'm trying to create a Webots simulation where I want my robot to speak when the Kinect camera detects a person.

I have a Kinect V2 hooked up to the USB port and it can detect a person on its own by running my Python code using PyKinect2 and pygame.

As the next step, I put that code into the Webots environment and I've added a robot to speak to the user when Kinect detects them. However, when the Kinect starts running and the window pops up, Webots clock stops ticking and the robot doesn't do anything. After I close the Kinect window, the robot speaks the message but that code should have been executed inside the Kinect code, as listed below.

I believe this might be a synchronization issue due to Kinect and Webots having their own clocks, but I'm not sure. Even if that's the case, I don't know how to proceed. Any advice is welcome.

Here are the relevant parts of my code, I can provide the full code if needed. The Kinect body detection is a slightly modified version of this example:

class BodyGameRuntime(object):

    # ----- other functions here -----

    def run(self):
        # -------- Main Program Loop -----------
        while not self._done:

            # --- frame and window resize logic here

            # --- Draw skeletons to _frame_surface
            if self._bodies is not None:

                # ---------- Body Found ----------
                speaker.speak("User detected!", volume=1.0) # Speak to the user

                for i in range(0, self._kinect.max_body_count):
                    body = self._bodies.bodies[i]
                    if not body.is_tracked: 
                        continue

                    joints = body.joints
                    # convert joint coordinates to color space 
                    joint_points = self._kinect.body_joints_to_color_space(joints)
                    self.draw_body(joints, joint_points, SKELETON_COLORS[i])

            # --- copy back buffer logic here

            # --- Update the screen with what we've drawn.
            pygame.display.update()
            pygame.display.flip()

            # --- Limit to 30 frames per second
            self._clock.tick(30)

        # Close our Kinect sensor, close the window and quit.
        self._kinect.close()
        pygame.quit()

robot = Robot()
speaker = Speaker("Speaker")
#timestep = int(robot.getBasicTimeStep())

__main__ = "Kinect"
game = BodyGameRuntime()
game.run()

Answer 1

Webots needs that you call the robot.step(timeStep) function on a regular basis in order to progress on the simulation time, otherwise it will simply stop the simulation and wait until the next call to robot.step(timeStep). I would simply add a call to robot.step(timeStep) in your main loop, right before self._clock.tick(30) and uncomment the initialization of timeStep in your main program.

Note that if your Webots simulation is running in real time, calling robot.step(X) will last approximately X milliseconds. So you should probably set the WorldInfo.basicTimeStep to 30 milliseconds in your world file and get rid of the call to self._clock.tick(30).