Add Curl query to Elastic cloud watcher

Question

I want to create following alert in Elastic cloud:

• Raise alert if number of unallocated shards exceeds 'x' value.

According to elastic-cloud documentation we can use the following query to Inspect unallocated shards:

GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state

Can this query be implemented using a elastic watcher? I have tried to implement using following code. I am unable to figure out the condition part:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "https",
        "host": "some-host",
        "port": 9243,
        "method": "get",
        "path": "/_cluster/stats",
        "params": {},
        "headers": {},
        "auth": {
          "basic": {
            "username": "user-name",
            "password": "some password"
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "def nodes = GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state {if (nodes > 300) {return true;}} return false;",
      "lang": "painless"
    }
  },
  "actions": {
    "send_email_alert": {
      "email": {
        "profile": "standard",
        "to": [
          "[email protected]"
        ],
        "subject": "Shard Alert",
        "body": {
          "text": "The number of shards per node exceeds 300. Please investigate."
        }
      }
    }
  }
}

I am new to elastic cloud and have reached an impasse.

Answer 1

You can use _cluster/health API call to see the number of unassigned shards. Here is all available watcher HTTP input if needed.

You can use ctx to access and use the value of any output. In your case we can use ctx.payload.unassigned_shards.

"condition": {
  "compare": {
    "ctx.payload.unassigned_shards": {
      "gt": 4
    }
  }
}

You can find more information about ctx and how watcher works? in the following link. https://www.elastic.co/guide/en/elasticsearch/reference/current/how-watcher-works.html

Here is the full example:

POST _watcher/watch/_execute
{
  "watch": {
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "https",
        "host": "cluster-id",
        "port": 9243,
        "method": "get",
        "path": "/_cluster/health",
        "params": {},
        "headers": {},
        "auth": {
          "basic": {
            "username": "::es_redacted::",
            "password": "::es_redacted::"
          }
        }
      }
    }
  },
    "condition": {
      "compare": {
        "ctx.payload.unassigned_shards": {
          "gt": 4
        }
      }
    },
  "actions": {
    "send_email_alert": {
      "email": {
        "profile": "standard",
        "to": [
          "::es_redacted::"
        ],
        "subject": "Shard Alert",
        "body": {
          "text": "The number of unassigned shards is {{ctx.payload.unassigned_shards}} and it exceeded 3. "
        }
      }
    }
  }
  }
}

Here is the email as output:

The number of unassigned shards is 5 exceeded 3.

---

Important note: Please control your ES_URL and make sure you can see the expected output with curl command before working on watcher. Eg.

musab@musab-mac Desktop % curl -k "https://your_cluster_name.es.us-east-2.aws.elastic-cloud.com:9243/_cluster/health?pretty" -u username:password
{
  "cluster_name" : "0ce67bce635a4b3882c580678b5cb4f5",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 720,
  "active_shards" : 720,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.3103448275862
}