How to combine items from a Dictionary into a list of dictionaries based on matching key/value details

Question

I have a list of dictionaries

    "tor_vlans": [
        {
            "switch-ls01": {
                "Vlan1": {
                    "id": "1",
                    "ip": "unassigned",
                    "ok": "NO",
                    "protocol": "down",
                    "status": "up"
                },
                "Vlan10": {
                    "id": "10",
                    "ip": "10.10.10.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up"
                },
                "Vlan20": {
                    "id": "20",
                    "ip": "10.10.20.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up"
                }
            }
        }
    ]

Using the ansible.utils.ipaddr module I've extracted the ip values from the above and created a new dictionary containing additional details for each subnet

    "subnet_details": {
        "10.10.10.2/24": {
            "dhcp_scope": "10.10.10.4-10.10.10.254",
            "tor_1_ip": "10.10.10.2/24",
            "tor_2_ip": "10.10.10.3/24",
            "vrrp": "10.10.10.1/24"
        },
        "10.10.20.2/24": {
            "dhcp_scope": "10.10.20.4-10.10.20.254",
            "tor_1_ip": "10.10.20.2/24",
            "tor_2_ip": "10.10.20.3/24",
            "vrrp": "10.10.20.1/24"
        }
    }

I'm attempting to combine both sets of dictionaries into a new list of dictionaries by matching the ip value from the tor_vlans list with the keys from subnet_details

Using Jinja2 templating below

    - name: Combine both based on matching ips and keys
      ansible.builtin.set_fact:
        my_new_dict: |
          {% for tor in tor_vlans %}
          {% for key,value in tor.items() %}
          {% for vlan,facts in value.items() %}
          {% for cidr,param in subnet_details.items() %}
          {% if cidr in facts %}
          {{ key }}:
            {{ vlan }}:
              {{ facts }}
              {{ param }}
          {% else %}
          {{ key }}:
            {{ vlan }}:
              {{ facts }}
          {% endif %}
          {% endfor %}
          {% endfor %}
          {% endfor %}
          {% endfor %}

The expected output would be

    "my_new_dict": [
        {
            "switch-ls01": {
                "Vlan1": {
                    "id": "1",
                    "ip": "unassigned",
                    "ok": "NO",
                    "protocol": "down",
                    "status": "up"
                },
                "Vlan10": {
                    "id": "10",
                    "ip": "10.10.10.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up",
                    "dhcp_scope": "10.10.10.4-10.10.10.254",
                    "tor_1_tep_ip": "10.10.10.2/24",
                    "tor_2_tep_ip": "10.10.10.3/24",
                    "vrrp": "10.10.10.1/24"
                },
                "Vlan20": {
                    "id": "20",
                    "ip": "10.10.20.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up",
                    "dhcp_scope": "10.10.20.4-10.10.20.254",
                    "tor_1_tep_ip": "10.10.20.2/24",
                    "tor_2_tep_ip": "10.10.20.3/24",
                    "vrrp": "10.10.20.1/24"
                }
            }
        }
    ]

The actual output that i'm getting is

    "my_new_dict|from_yaml": {
        "switch-ls01": {
            "Vlan20": {
                "id": "20",
                "ip": "10.10.20.2/24",
                "ok": "YES",
                "protocol": "up",
                "status": "up"
            }
        }
    }

If there's a way of combing other than using Jinja, please let me know.

Likewise, if the subnet details could be created directly in tor_vlans using the ansible.utils.ipaddr module, I can't figure it out.

The "Vlan1", "Vlan10", etc... that are causing me issues when trying to use the ipaddr module directly on tor_vlans as these keys are dynamically learnt so i can't presume they will always be the same, i.e. sometimes they could be "Vlan25", "Vlan55", etc...

I've found a good few posts on this for Python but, can't seem to find any similar Ansible posts

Answer 1

Create list of updates

    update: |
      {% filter from_yaml %}
      {% for vlan in tor_vlans %}
      {% for sw,vlans in vlan.items() %}
      [ {{ sw }}: {
      {% for k,v in vlans.items() %}
      {% for s,d in subnet_details.items() %}
      {% if [v.ip]|ansible.utils.ipaddr(s)|length > 0 %}
      {{ k }}: {{ v|combine(d) }},
      {% endif %}
      {% endfor %}
      {% endfor %} },
      {% endfor %} ]
      {% endfor %}
      {% endfilter %}

gives

  update:
  - switch-ls01:
      Vlan10:
        dhcp_scope: 10.10.10.4-10.10.10.254
        id: '10'
        ip: 10.10.10.2/24
        ok: 'YES'
        protocol: up
        status: up
        tor_1_ip: 10.10.10.2/24
        tor_2_ip: 10.10.10.3/24
        vrrp: 10.10.10.1/24
      Vlan20:
        dhcp_scope: 10.10.20.4-10.10.20.254
        id: '20'
        ip: 10.10.20.2/24
        ok: 'YES'
        protocol: up
        status: up
        tor_1_ip: 10.10.20.2/24
        tor_2_ip: 10.10.20.3/24
        vrrp: 10.10.20.1/24

zip the lists and combine the items

    my_new_dict: "{{ tor_vlans |
                     zip(update) |
                     map('combine', recursive=True) }}"

gives

  my_new_dict:
  - switch-ls01:
      Vlan1:
        id: '1'
        ip: unassigned
        ok: 'NO'
        protocol: down
        status: up
      Vlan10:
        dhcp_scope: 10.10.10.4-10.10.10.254
        id: '10'
        ip: 10.10.10.2/24
        ok: 'YES'
        protocol: up
        status: up
        tor_1_ip: 10.10.10.2/24
        tor_2_ip: 10.10.10.3/24
        vrrp: 10.10.10.1/24
      Vlan20:
        dhcp_scope: 10.10.20.4-10.10.20.254
        id: '20'
        ip: 10.10.20.2/24
        ok: 'YES'
        protocol: up
        status: up
        tor_1_ip: 10.10.20.2/24
        tor_2_ip: 10.10.20.3/24
        vrrp: 10.10.20.1/24

---

Example of a complete playbook for testing

- hosts: all

  vars:

    tor_vlans:
      - switch-ls01:
          Vlan1:
            id: '1'
            ip: unassigned
            ok: 'NO'
            protocol: down
            status: up
          Vlan10:
            id: '10'
            ip: 10.10.10.2/24
            ok: 'YES'
            protocol: up
            status: up
          Vlan20:
            id: '20'
            ip: 10.10.20.2/24
            ok: 'YES'
            protocol: up
            status: up

    subnet_details:
      10.10.10.2/24:
        dhcp_scope: 10.10.10.4-10.10.10.254
        tor_1_ip: 10.10.10.2/24
        tor_2_ip: 10.10.10.3/24
        vrrp: 10.10.10.1/24
      10.10.20.2/24:
        dhcp_scope: 10.10.20.4-10.10.20.254
        tor_1_ip: 10.10.20.2/24
        tor_2_ip: 10.10.20.3/24
        vrrp: 10.10.20.1/24

    update: |
      {% filter from_yaml %}
      {% for vlan in tor_vlans %}
      {% for sw,vlans in vlan.items() %}
      [ {{ sw }}: {
      {% for k,v in vlans.items() %}
      {% for s,d in subnet_details.items() %}
      {% if [v.ip]|ansible.utils.ipaddr(s)|length > 0 %}
      {{ k }}: {{ v|combine(d) }},
      {% endif %}
      {% endfor %}
      {% endfor %} },
      {% endfor %} ]
      {% endfor %}
      {% endfilter %}

    my_new_dict: "{{ tor_vlans |
                     zip(update) |
                     map('combine', recursive=True) }}"

  tasks:

    - debug:
        var: update
    - debug:
        var: my_new_dict

Answer 2

I've found a good few posts on this for Python

Often it makes sense to perform more complex data manipulation tasks in Python. Writing a custom filter module is very simple; if we were to drop the following code into filter_plugins/vlans.py:

import netaddr


def filter_update_vlans(v):
    for block in v:
        for switchname, vlans in block.items():
            for vlanname, vlandata in vlans.items():
                try:
                    addr = netaddr.IPNetwork(vlandata["ip"])
                    vlandata["vrrp"] = f"{addr[1]}/{addr.prefixlen}"
                    vlandata["tor_1_tep_ip"] = f"{addr[2]}/{addr.prefixlen}"
                    vlandata["tor_2_tep_ip"] = f"{addr[3]}/{addr.prefixlen}"
                    vlandata["dhcp_scope"] = f"{addr[4]}-{addr[-2]}"
                except netaddr.AddrFormatError:
                    continue

    return v


class FilterModule:
    def filters(self):
        return {
            "update_vlans": filter_update_vlans,
        }

We could reduce our playbook to:

- hosts: localhost
  gather_facts: false

  tasks:
    - set_fact:
        tor_vlans: "{{ tor_vlans | update_vlans }}"

    - debug:
        var: tor_vlans

Which, given your sample data, produces as output:

TASK [debug] *******************************************************************
ok: [localhost] => {
    "tor_vlans": [
        {
            "switch-ls01": {
                "Vlan1": {
                    "id": "1",
                    "ip": "unassigned",
                    "ok": "NO",
                    "protocol": "down",
                    "status": "up"
                },
                "Vlan10": {
                    "dhcp_scope": "10.10.10.4-10.10.10.254",
                    "id": "10",
                    "ip": "10.10.10.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up",
                    "tor_1_tep_ip": "10.10.10.2/24",
                    "tor_2_tep_ip": "10.10.10.3/24",
                    "vrrp": "10.10.10.1/24"
                },
                "Vlan20": {
                    "dhcp_scope": "10.10.20.4-10.10.20.254",
                    "id": "20",
                    "ip": "10.10.20.2/24",
                    "ok": "YES",
                    "protocol": "up",
                    "status": "up",
                    "tor_1_tep_ip": "10.10.20.2/24",
                    "tor_2_tep_ip": "10.10.20.3/24",
                    "vrrp": "10.10.20.1/24"
                }
            }
        }
    ]
}