Icinga2 event plugin command starting a rundeck job via api

Question

i made myself a test environment in icinga2 with a tomcat server. I would like to combine the two softwares rundeck and icinga. My idea is to start a rundeck job, when icinga detects a problem. In my case I have a tomcat server, where i fill up the swap memory, which should start the rundeck job to clear the swap. I am using the Icinga2 Director for managing. I created an event plugin command, which should execute the rundeck api command as a script, called "rundeckapi". It looks like this:

#/usr/lib64/nagios/plugins/rundeckapi
#!/bin/bash 
    curl --location --request POST 'rundeck-server:4440/api/38/job/9f04657a-eaab-4e79-a5f3-00d3053f6cb0/run' \
    --header 'X-Rundeck-Auth-Token: GuaoD6PtH5BhobhE3bAPo4mGyfByjNya' \
    --header 'Content-Type: application/json' \
    --header 'Cookie: JSESSIONID=node01tz8yvp4gjkly8kpj18h8u5x42.node0' \
    --data-raw '{
        "options": {
            "IP":"192.168.4.13"
        }
    }'

(I also tried to just paste the command in the command field in the director, but this didn't work either.)

I placed it in the /usr/lib64/nagios/plugins/ directory and set the configuration in icinga for the command as following:

#zones.d/director-global/command.conf

    object EventCommand "SWAP clear" {
        import "plugin-event-command"
        command = [ PluginDir + "/rundeckapi" ]
    }

The service template looks like this:

#zones.d/master/service_templates.conf

        template Service "SWAP" {
        check_command = "swap"
        max_check_attempts= "5"
        check_interval = 1m
        retry_interval = 15s
        check_timeout = 10s
        enable_notifications = true
        enable_active_checks = true
        enable_passive_checks = true
        enable_event_handler = true
        enable_flapping = true
        enable_perfdata = true
        event_command = "SWAP clear"
        command_endpoint = host_name
    }

Then I added the service to the host.

I enabled the debug mode and started to fill the SWAP and watched at the debug.log, with tail -f /var/log/icinga2/debug.log | grep 'event handler' and found this:

notice/Checkable: Executing event handler 'SWAP clear' for checkable 'centos_tomcat_3!SWAP'

The centos_tomcat_3 is the host for testing. IT seems like the event handler is executing the the script, but when I look at the rundeck server, i can't find a running job. When i start the rundeckapi script manually it works and i can see the job on rundeck.

I already read the documentation from icinga, but i didn't help.

I would be very thankful if someone could help me. Thanks in advance.

Answer 1

Define the plugin as an event handler and assign it to the host.

I tested using this docker environment modified with Rundeck official image + a NGINX host:

version: '2'
services:
  icinga2:
    #image: jordan/icinga2
    build:
      context: ./
      dockerfile: Dockerfile
    restart: on-failure:5
    # Set your hostname to the FQDN under which your
    # sattelites will reach this container
    hostname: icinga2
    env_file:
      - secrets_sql.env
    environment:
      - ICINGA2_FEATURE_GRAPHITE=1
      # Important:
      # keep the hostname graphite the same as
      # the name of the graphite docker-container
      - ICINGA2_FEATURE_GRAPHITE_HOST=graphite
      - ICINGA2_FEATURE_GRAPHITE_PORT=2003
      - ICINGA2_FEATURE_GRAPHITE_URL=http://graphite
      # - ICINGA2_FEATURE_GRAPHITE_SEND_THRESHOLDS=true
      # - ICINGA2_FEATURE_GRAPHITE_SEND_METADATA=false
      - ICINGAWEB2_ADMIN_USER=admin
      - ICINGAWEB2_ADMIN_PASS=admin
      #- ICINGA2_USER_FULLNAME=Icinga2 Docker Monitoring Instance
      - DEFAULT_MYSQL_HOST=mysql
    volumes:
      - ./data/icinga/cache:/var/cache/icinga2
      - ./data/icinga/certs:/etc/apache2/ssl
      - ./data/icinga/etc/icinga2:/etc/icinga2
      - ./data/icinga/etc/icingaweb2:/etc/icingaweb2
      - ./data/icinga/lib/icinga:/var/lib/icinga2
      - ./data/icinga/lib/php/sessions:/var/lib/php/sessions
      - ./data/icinga/log/apache2:/var/log/apache2
      - ./data/icinga/log/icinga2:/var/log/icinga2
      - ./data/icinga/log/icingaweb2:/var/log/icingaweb2
      - ./data/icinga/log/mysql:/var/log/mysql
      - ./data/icinga/spool:/var/spool/icinga2
      # Sending e-mail
      #  See: https://github.com/jjethwa/icinga2#sending-notification-mails
      #  If you want to enable outbound e-mail, edit the file mstmp/msmtprc
      #  and configure to your corresponding mail setup. The default is a
      #  Gmail example but msmtp can be used for any MTA configuration.
      #  Change the aliases in msmtp/aliases to your recipients.
      #  Then uncomment the rows below
      # - ./msmtp/msmtprc:/etc/msmtprc:ro
      # - ./msmtp/aliases:/etc/aliases:ro
    ports:
      - "80:80"
      - "443:443"
      - "5665:5665"
  graphite:
    image: graphiteapp/graphite-statsd:latest
    container_name: graphite
    restart: on-failure:5
    hostname: graphite
    volumes:
      - ./data/graphite/conf:/opt/graphite/conf
      - ./data/graphite/storage:/opt/graphite/storage
      - ./data/graphite/log/graphite:/var/log/graphite
      - ./data/graphite/log/carbon:/var/log/carbon
  mysql:
    image: mariadb
    container_name: mysql
    env_file:
      - secrets_sql.env
    volumes:
      - ./data/mysql/data:/var/lib/mysql
      # If you have previously used the container's internal DB use:
      #- ./data/icinga/lib/mysql:/var/lib/mysql
  rundeck:
    image: rundeck/rundeck:3.3.12
    hostname: rundeck
    ports:
      - '4440:4440'
  nginx:
    image: nginx:alpine
    hostname: nginx
    ports:
      - '81:80'

Rundeck side:

1. To access Rundeck open a new tab in your browser using the http://localhost:4440 web address. You can access with user: admin and password: admin.

2. Create a new project and create a new job, I created the following one, you can import it to your instance:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: c3e0860c-8f69-42f9-94b9-197d0706a915
  loglevel: INFO
  name: RestoreNGINX
  nodeFilterEditable: false
  options:
  - name: opt1
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - exec: echo "hello ${option.opt1}"
    keepgoing: false
    strategy: node-first
  uuid: c3e0860c-8f69-42f9-94b9-197d0706a915

3. Now go to the User Icon (up to right) > Profile, now click on the + icon ("User API Tokens" section) and save the API key string, useful to create the API call script from the Icinga2 container.

4. Go to the Activity page (left menu) and click on the "Auto Refresh" checkbox.

Incinga2 side:

1. You can enter Icinga 2 by opening a new tab in your browser using the http://localhost URL, I defined username: admin and password: admin in the docker-compose file.

2. Add the following script as a command at /usr/lib/nagios/plugins path with the following content (it's a curl api call like your scenario, the API key is the same generated in the third step from "Rundeck side" section of this step-by-step):

#!/bin/bash 
curl --location --request POST 'rundeck:4440/api/38/job/c3e0860c-8f69-42f9-94b9-197d0706a915/run' --header 'X-Rundeck-Auth-Token: Zf41wIybwzYhbKD6PrXn01ZMsV2aT8BR' --header 'Content-Type: application/json' --data-raw '{ "options": { "opt1": "world" } }'

Also make the script executable: chmod +x /usr/lib/nagios/plugin/restorenginx

3. In the Icinga2 browser tab, go to the Icinga Director (Left Menu) > Commands. On the "Command Type" list select "Event Plugin Command", on the "Command Name" textbox type "restorenginx" and on the "Command" textbox put the full path of the script (/usr/lib/nagios/plugins/restorenginx). Then click on the "Store" button (bottom) and now click on Deploy (up).

Check how looks.

This is the config preview (at zones.d/director-global/commands.conf):

object EventCommand "restorenginx" {
    import "plugin-event-command"
    command = [ "/usr/lib/nagios/plugins/restorenginx" ]
}

4. Now, create the host template (In my example I'm using an Nginx container to monitoring), go to Icinga Director (Left Menu) > Hosts, and select "Host Templates". Then click on the + Add link (up). On the Name type the host template name, I used "nginxSERVICE", on the "check command" textbox put the command to check the host alive (I used "ping"). Now in the Event command textbox select the Command created in the previous step.

Check how looks.

5. Now it's time to create the host (based on the previous steps template). Go to Icinga Direcrector (Left Menu) > Hosts and select "Host". Then click on the + Add link (up). On the hostname type the server hostname (nginx, defined on the docker-compose file), In "Imports" select the template is created in the previous step ("nginxSERVICE"), type anything on the "Display name" textbox, and in the "Host address" add the Nginx container IP. Click on the "Store" button" and then on the "Deploy" link at the top.

Check how looks.

6. To enable the Event Hander on the host, go to the "Overview" (Left menu) > Hosts, select "NGINX", scroll down on the right section and enable "Event Handler" on the "Feature Commands" section.

Check how looks.

Nginx side (it's time to test the script):

1. Stop the container and go to the Rundeck Activity page browser tab, you'll see the job launched by the Icinga2 monitoring tool.