Wait for multiple lines to be logged in Ansible

Question

I am running an Ansible script on EC2 with Cloudformation. The ansible script runs when the instance is created and there's an autoscaling group waiting for cfn-signal from the instance.

I want Ansible to wait for few logs to appear. The logs can come in any particular order saying that a module has finished being deployed. After all logs have appeared, I would like to signal the CF stack that the application is healthy.

I have tried the following:

- name: Check if applications were deployed properly
  wait_for:
    path: "{{ LOGFILE }}"
    search_regex: "{{ item }} has finished"
    timeout: 120
    delay: 3
  loop:
    - application1
    - application2
    - application3

- name: Send Cloudformation sucess signal
  command: "/opt/aws/bin/cfn-signal --stack {{ stackname }} --resource AutoScalingGroup --region {{ region }} --success true"

However, this is not working and it seems like Ansible is waiting in order. How can I achieve this asynchronously?

Answer 1

For a test file cleanandempty.log

START
app4 has finished
line
app2 has finished
another line
app3 has finished
again a line
app1 has finished
END

a minimal example playbook

---
- hosts: localhost
  become: false
  gather_facts: false

  vars:

    LOGFILE: cleanandempty.log

  tasks:

  - name: Run code which stare at log file
    shell:
      cmd: grep -e 'app2 has finished' -e 'app3 has finished' -e 'app1 has finished' -o {{ LOGFILE }} | wc -l
    register: result
    until: result.stdout | int == 3
    retries: 100
    delay: 1

would result into an output of

PLAY [localhost] ******************************************************
FAILED - RETRYING: Run code which stare at log file (100 retries left).
FAILED - RETRYING: Run code which stare at log file (99 retries left).
FAILED - RETRYING: Run code which stare at log file (98 retries left).
FAILED - RETRYING: Run code which stare at log file (97 retries left).
FAILED - RETRYING: Run code which stare at log file (96 retries left).
FAILED - RETRYING: Run code which stare at log file (95 retries left).
FAILED - RETRYING: Run code which stare at log file (94 retries left).

TASK [Run code which stare at log file] *******************************
changed: [localhost]

as soon as the last application startup has been logged.

In order to find multiple patterns which are spread randomly over multiple lines Matching Control grep -e was used and the amount of occurrences were counted.

Answer 2

you can use parameter async, e.g.

    - hosts: localhost
      gather_facts: false
      tasks:
        - name: Check if applications were deployed properly
          shell: "until cat test.txt | grep -q '{{item}} has finished'; do  sleep 3; done"
          register: log_output
          async: 35
          poll: 0
          loop:
            - application1
            - application2
            - application3

        - name: Check sync status
          async_status:
            jid: "{{ async_result_item.ansible_job_id }}"
          loop: "{{ log_output.results }}"
          loop_control:
          loop_var: "async_result_item"
          register: async_poll_results
          until: async_poll_results.finished
          retries: 30

Source: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_async.html

Edit: I've missed the unordered need so I've changed my answer accordingly

Answer 3

Define an inventory where hosts are represented by applications. This can be made dynamically in the playbook, but the static one is simpler and could be useful in future:

applications:
  vars:
    ansible_connection: local # for testing
  hosts:
    app1:
      name: application1
      log_file: "path/to/the/log/file.log"
    app2:
      name: application2
      # ... same for other apps

Then adjust your playbook a bit. Remove the loop and run the play on application group of hosts:

- hosts: applications
  tasks:
    - name: Check if applicationa were deployed properly
      wait_for:
        path: "{{ log_file }}"
        search_regex: "{{ name }} has finished"
        timeout: 120
        delay: 3

This way you can run it both locally and remotely. Please note that the concerns on false positives expressed by @U880D in the comments are still valid. Also, if your log files are empty upon the playbook startup, you will likely run into a ValueError: cannot map an empty file error, that should be addressed additionally.