.count aggregate significantly slows execution time

Question

I am refactoring some old code to hope to speed up and make information more secure. In order to do so I am trying to use the MongoDB aggregation framework to get all users near a specific location. Originally this was being solved by handling some spherical calculations on the front end but to do so it was sending lat, lng coordinates of each user back to the front end which is insecure and exposes location data of users un-necessarily.

My solution is use the aggregation framework and $geoNear to get the list of nearby users and then also populate a field with the distance in miles called distanceAway. Everything is working well until I try to set up pagination. The $count stage slows the execution of the route dramatically.

Below is my code:

module.exports = async function findUsersNearLocationAggregate(baseLocation, page = 1, limit = 20) {
    // quick validation on location object passed
    if(!lodash.get(baseLocation, 'geometry.location', undefined)) 
        throw new Error('NO_LOCATION_SPECIFIED')

    //  Grab user location to use as location origin
    const { lat, lng } = baseLocation.geometry.location

    let query = {
        $geoNear: {
            near: { type: "Point", coordinates: [lng, lat] },
            distanceField: "distanceAway",
            spherical: true,
            distanceMultiplier: CONVERT_METERS_TO_MILES
        }
    }

    const users = await User.aggregate([
        query,
        { $skip: ((page - 1) * limit) },
        { $limit: limit }
    ])

    // const [{ count }] = await User.aggregate([
        // query,
        // { $count: 'count' }
    // ])

    return {
        user: users,                    
        // totalPages: Math.ceil(count / limit)
        // currentPage: page
    }
}

This function was meant to return the list of users (limit of 20 at a time) and show other data such as totalPages and currentPage to track the pagination on the front-end and make subsequent requests.

When I comment out the following line:

    const [{ count }] = await User.aggregate([
        query,
        { $count: 'count' }
    ])

the execution of the route that uses the call is max 100ms. When I comment it in the call jumps to approx 1100ms.

Is there someway to get the pagination data I am looking for without a significant increase in request time? 10X seems quite large.