Can't access Google Fit sleep data. Always 403

Question

I'm trying to access Google Fit api sleep data. No matter what I try I always get a 403 stating that I don't haver permission. I've looked into documentation and I've added all Fit scopes into the Oauth playground and I still have the same response

I use the following body:

{
    "aggregateBy": [
        {
            "dataSourceId": "com.google.sleep.segment"
        }
    ],
    "bucketByTime": {
        "durationMillis": 300000
    },
    "startTimeMillis": 1707609600000,
    "endTimeMillis": 1707689799000
}

And I always get a 403:

{
  "error": {
    "status": "PERMISSION_DENIED", 
    "message": "The caller does not have permission", 
    "code": 403, 
    "errors": [
      {
        "reason": "forbidden", 
        "message": "The caller does not have permission", 
        "domain": "global"
      }
    ]
  }
}

Which scope am I missing? What am I doing wrong. All other fitness related requests work without an issue