Replaying mouse movements with DispatchQueue is laggy

Question

I've recorded a mouse movement (the movement file is here) in the form of (x,y,delay_in_ms) array and I'm trying to replay it back like this:

for movement in movements {
    DispatchQueue.global().asyncAfter(deadline: .now() + .milliseconds(Int(movement.time))) {
        let point = CGPoint(x: Int(movement.x), y: Int(movement.y))
        let moveEvent = CGEvent(mouseEventSource: nil, mouseType: .mouseMoved, mouseCursorPosition: point, mouseButton: .left);
        moveEvent!.post(tap: CGEventTapLocation.cghidEventTap)
    }
}

And it kind of works, but it's very laggy. I only see a couple of points from the shape. On the other hand, when I replay the same file with robotjs:

const path = await fs.readJSON("/Users/zzzz/path.txt");

await Promise.all(path.map((move: {x: number, y: number, time: number}) => {
    return new Promise<void>((resolve) => {
        setTimeout(async () => {
            const x = move.x;
            const y = move.y;

            robot.moveMouse(x, y);
            resolve();
        }, move.time);
    })
}));

The movement is silky smooth, exactly the way I recorded it.

I've tried replacing the DispatchQueue in Swift with a bunch of usleep calls. Then I could see all the points in the shape, but it was just slow, maybe 4x slower than intended.

Is there anything wrong with my usage of DispatchQueue?

Answer 1

A few thoughts:

• Consider using the main queue, rather than a global queue. You don’t really want to be initiating UI actions from a background thread.

• I would not expect it to kick in quite at this point, but be aware that scheduling multiple timers in the future can get caught up in “timer coalescing”, a power saving feature whereby events will be bunched together to reduce battery consumption. The general rule is that if two independently scheduled timers are with 10% of each other, they can be coalesced.

  Try GCD timers (rather than asyncAfter) with the .strict option. Or rather than scheduling all up front, schedule the first one and have it schedule the next one, recursively.

• I notice that many of the points are ~7.5msec apart. That's faster than 120 events per second (“eps”). The event handler might not be able to handle them that quickly and you might see points dropped. I'm not sure where the precise cutoff is, but in my test, it had no problem with 60eps, but at 120eps I would see the occasional dropped event, and I'm sure that at faster than 120 eps, you may see even more dropped.

Answer 2

@Rob makes some good points here. You 100% need to be posting UI events on the main thread (or queue). If you need tight control over timing, I would suggest using the "User Interactive" Quality of Service class. (https://developer.apple.com/library/archive/documentation/Performance/Conceptual/EnergyGuide-iOS/PrioritizeWorkWithQoS.html)

JavaScript is a single threaded runtime, that largely pre-supposes that you're working in a web-browser (and thus is always user-interactive) context. With that in mind, it's not surprising that it's smoother than default-class GCD blocks.