Elastic Cloud Circuit_breaking_exception

Question

We recently upgraded our elastic cloud deployment from 6.8.5 to 7.9 After the upgrade we are seeing the following error time to time.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[parent] Data too large, data for [<http_request>] would be [416906520/397.5mb], which is larger than the limit of [408420352/389.5mb], real usage: [416906520/397.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=32399/31.6kb, in_flight_requests=0/0b, model_inference=0/0b, accounting=4714192/4.4mb]",
        "bytes_wanted" : 416906520,
        "bytes_limit" : 408420352,
        "durability" : "PERMANENT"
      }
    ],
    "type" : "circuit_breaking_exception",
    "reason" : "[parent] Data too large, data for [<http_request>] would be [416906520/397.5mb], which is larger than the limit of [408420352/389.5mb], real usage: [416906520/397.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=32399/31.6kb, in_flight_requests=0/0b, model_inference=0/0b, accounting=4714192/4.4mb]",
    "bytes_wanted" : 416906520,
    "bytes_limit" : 408420352,
    "durability" : "PERMANENT"
  },
  "status" : 429
}

This deployment consists of only one node with 1G memory. We would like to know the cause of this error. Is it due to the upgrade?

Thank you.

Answer 1

First, the circuit breaker is a protection that some request doesn't push your cluster over the limit of what it can handle — this is killing a single request rather than (potentially) the entire cluster. Also note that this HTTP request alone isn't too large, but it trips the parent circuit breaker — so this request on top of everything else would be too much.

The initial circuit breaker was already added in 6.2.0, but was tightened down further in 7.0.0. I assume that's the reason why you are seeing this (more frequently) now.

You could change the indices.breaker.total.limit, but this isn't a magic switch to get more out of your cluster. 1GB of memory might just not be enough for what you are trying to do.