Elasticsearch range query slow matching when combined with a specific filter

Question

I'm trying to figure out why our new cluster is slow and found some weird things.

When executing this:

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "reference_id": 4489 } },
        { "range": { "created_at": { "gte": "2020-03-15" }} }
      ]
    }
  }
}

And check the profiling, it looks like it's spending a lot of time in "match":

IndexOrDocValuesQuery
created_at:[1584230400000 TO 9223372036854775807]
match, 2.3s, 84.2%

If running the following query, which filters on another field (same mapping) it doesn't use any match at all:

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "root_id": 3413 } },
        { "range": { "created_at": { "gte": "2020-03-15" }} }
      ]
    }
  }
}

Now it just spends some time in advance (match is 0):

IndexOrDocValuesQuery
created_at:[1584230400000 TO 9223372036854775807]
advance, 376.5ms, 82.5%

Both root_id and reference_id are mapped as long (will change to keyword as I think it will be faster at least).

Now, the two queries are filtering on different fields, but still the same type of fields, why would it do a slooow "match" in the first query? Any ideas? Can I figure that out somehow? Also, why is it doing a "match" at all? I only do filtering so no scoring should be done, right?