Custom sort list based on another

Question

I'm trying to sort a list of files based on a desired extension preference. Let's say I have a list that looks like:

- name: Declare all assets
  ansible.builtin.set_fact:
    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum

Now, I'd like to get sort them in some order which could be changed by prompt, say: ['deb', 'appimage', 'tar.gz', '']. In this case '' means no extension, which in this case means it's a linux executable. Regarding the files that don't match the extension, I'm not intereseted in them, I'd prefer them to be removed.

Is this possible to do without writing a custom filter? I had the idea to prepend the extensions to the items with something like: map('regex_replace', '^(.*\\.)(deb|appimage|tar\\.gz)$', '\\2 \\1\\2') but I would still need to prepend a number or something to pass it through the regular ansible sort before trying to remove it to get the final filename. The entire thing looks very complicated and hard to read in that case and I don't have a good solution for no extension.

Answer 1

Given the lists of the files and the extensions

    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum
    asset_order: [deb, appimage, tar.gz]

Reverse the items

    asset_list_reverse: "{{ asset_list | map('reverse') }}"
    asset_order_reverse: "{{ asset_order | map('reverse') }}"

Use the fact that the test match:

succeeds if it finds the pattern at the beginning of the string

and create the ordered list of files with extensions

  asset_list_order: |
    {% filter flatten %}
    [{% for ext in asset_order_reverse %}
    {{ asset_list_reverse | select('match', ext) | map('reverse') }},
    {% endfor %}]
    {% endfilter %}

gives

  asset_list_order:
  - artifact.deb
  - artifact.appimage
  - artifact_amd64.tar.gz
  - artifact_arm64.tar.gz

Create a list of executables

  asset_list_exe: "{{ asset_list | reject('search', '\\.') }}"

gives

  asset_list_exe:
  - artifact_amd64

Add the lists

  result: "{{ asset_list_order + asset_list_exe }}"

gives

  result:
  - artifact.deb
  - artifact.appimage
  - artifact_amd64.tar.gz
  - artifact_arm64.tar.gz
  - artifact_amd64

---

Notes

1. Example of a complete playbook for testing

- hosts: localhost

  vars:

    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum
    asset_list_reverse: "{{ asset_list | map('reverse') }}"

    asset_order: [deb, appimage, tar.gz]
    asset_order_reverse: "{{ asset_order | map('reverse') }}"

    asset_list_order: |
      {% filter flatten %}
      [{% for ext in asset_order_reverse %}
      {{ asset_list_reverse | select('match', ext) | map('reverse') }},
      {% endfor %}]
      {% endfilter %}
    asset_list_exe: "{{ asset_list | reject('search', '\\.') }}"
    result: "{{ asset_list_order + asset_list_exe }}"

  tasks:

    - debug:
        var: asset_list_order
    - debug:
        var: asset_list_exe
    - debug:
        var: result

---

2. If you want to put the empty string into the list, for example,

  asset_order: [deb, appimage, tar.gz, '']

remove it from the list of the reversed extensions

  asset_order_reverse: "{{ asset_order | select | map('reverse') }}"

and test it when assembling the result

  result: "{{ ('' in asset_order) |
              ternary(asset_list_order + asset_list_exe,
                      asset_list_order) }}"

---

3. The next option is to create the complete list in Jinja. This keeps the order of the extensions including the executables. The example of a complete playbook for testing

- hosts: localhost

  vars:

    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum
    asset_order: [deb, '', appimage, tar.gz]

    asset_list_reverse: "{{ asset_list | map('reverse') }}"
    asset_order_reverse: "{{ asset_order | map('reverse') }}"

    result: |
      [{% for ext in asset_order_reverse %}
      {% if ext|length > 0 %}
      {{ asset_list_reverse | select('match', ext) | map('reverse') }},
      {% else %}
      {{ asset_list | reject('search', '\.') }},
      {% endif %}
      {% endfor %}]

  tasks:

    - debug:
        var: result | flatten

gives

  result | flatten:
  - artifact.deb
  - artifact_amd64
  - artifact.appimage
  - artifact_amd64.tar.gz
  - artifact_arm64.tar.gz

Answer 2

I have three answers for your question (scroll down to the real problem).

Answer to the original question

Basically, the idea could be the following:

• detect the available extensions
• create a mapping "full artifact name - extension" for simpler processing (otherwise, we would need to have a more complex loop)
• extract the artifacts that match the prompt list in a loop to keep the order.

An example playbook will look like this:

---
- name: Filter the assets based on the given list of extensions
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    sort_prompt: ['deb', 'appimage', 'tar.gz', '']
    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum
    asset_list_extensions: >
      {{ asset_list | map('split', '.', 1) | map(attribute='1', default='') }}
    asset_list_zipped: >
      {{ asset_list | zip(asset_list_extensions) }}
    asset_list_filtered: []
  tasks:
    - name: Filter the assets
      set_fact:
        asset_list_filtered: >
          {{ 
            asset_list_filtered 
              + asset_list_zipped | selectattr(1, 'equalto', item) 
          }}
      loop: "{{ sort_prompt }}"

    - name: Display the results
      vars:
        asset_list_filtered: "{{ asset_list_filtered | map(attribute='0') }}"
      debug:
        var: asset_list_filtered

This would give the following results:

ok: [localhost] => 
  asset_list_filtered:
  - artifact.deb
  - artifact.appimage
  - artifact_amd64.tar.gz
  - artifact_arm64.tar.gz
  - artifact_amd64

Some bits of explanation:

    # We will extract the available extensions:
    #   map('split', '.', 1) prepares a list of two items:
    #     - package name
    #     - package extension (e.g. all the remaining substring)
    #   map(attribute='1', default='') extracts the extension:
    #     attribute='1' is the same as some_list[1]
    #     default='' will cover a missing extension
    asset_list_extensions: >
      {{ asset_list | map('split', '.', 1) | map(attribute='1', default='') }}

    # Here we're merging the lists of artifact names and the extensions
    # to have a link between them
    asset_list_zipped: >
      {{ asset_list | zip(asset_list_extensions) }}

    # We will start to build the resulting list from here.
    # Could be replaced by `| default([]) in the template itself`
    asset_list_filtered: []

Solution for a case when the package name contains dots

In fact, that's even simpler than the original. But instead of filters, we will rely on regular expressions and Jinja2 templating:

- name: Filter the assets based on the given list of extensions
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    sort_prompt: [ deb, appimage, '', tar.xz, tar.gz ]
    asset_list:
      - artifact_amd64.tar.gz
      - artifact_amd64.tar.gz.sbom
      - artifact_arm64.tar.gz
      - artifact.deb
      - artifact.appimage
      - artifact_amd64
      - artifact_amd64.sha256sum
      - mise-v2024.3.1-linux-arm64
      - mise-v2024.3.1-linux-arm64-musl
      - mise-v2024.3.1-linux-arm64-musl.tar.gz
      - mise-v2024.3.1-linux-arm64-musl.tar.xz
      - mise-v2024.3.1-linux-arm64.tar.gz
      - mise-v2024.3.1-linux-arm64.tar.xz
      - mise-v2024.3.1-linux-armv6
      - mise-v2024.3.1-linux-armv6-musl
      - mise-v2024.3.1-linux-armv6-musl.tar.gz
      - mise-v2024.3.1-linux-armv6-musl.tar.xz
    asset_list_filtered: []
  tasks:
    - name: Filter the assets
      vars:
        package: mise
        regex_to_find_non_empty_extensions: '^{{ package }}.*({{ item }})$'
        regex_to_find_empty_extensions: '^{{ package }}$'
      set_fact:
        asset_list_filtered: >
          {{ 
            asset_list_filtered +
              asset_list 
              | map('regex_search', regex_to_find_non_empty_extensions) 
              | select            
              if item != ''
              else
                asset_list 
                | map('regex_search', regex_to_find_empty_extensions) 
                | select
          }}
      loop: "{{ sort_prompt }}"

    - name: List the results
      debug:
        var: asset_list_filtered

Now we will use the package variable to filter out the results:

• For package artifact:

  ansible-playbook playbook.yaml -e package=artifact
  

  asset_list_filtered:
  - artifact_amd64.tar.gz
  - artifact_arm64.tar.gz
  

• For package artifact_amd64 (note that at this point making the solution clever enough to distinct _ and - would be equal to reinventing an existing module):

  ansible-playbook playbook.yaml -e package=artifact_amd64
  

  asset_list_filtered:
  - artifact_amd64
  - artifact_amd64.tar.gz
  

• For package mise:

  ansible-playbook playbook.yaml -e package=mise
  

  Or even mise-v2024.3.1:

  ansible-playbook playbook.yaml -e package=mise-v2024.3.1
  

  asset_list_filtered:
  - mise-v2024.3.1-linux-arm64-musl.tar.xz
  - mise-v2024.3.1-linux-arm64.tar.xz
  - mise-v2024.3.1-linux-armv6-musl.tar.xz
  - mise-v2024.3.1-linux-arm64-musl.tar.gz
  - mise-v2024.3.1-linux-arm64.tar.gz
  - mise-v2024.3.1-linux-armv6-musl.tar.gz
  

Solution for the real problem behind the question

For your particular case, chances are that you don't need to implement this logic:

• Ansible has a built-in package module that is agnostic to the OS distribution and selects the package automatically;
• Ansible can gather the facts about the hosts that it is running on, including the OS and distribution name. Generally, one should rely on them instead of selecting the package extension manually. Check out Conditionals based on ansible_facts for details.
• If you're installing the releases from GitHub, you might want to look into quera.github.install_from_github module that can select the package type based on the discovered OS/arch facts passed via asset_regex.