How to use encrypted YAML inventory file with Ansible playbook?

Question

I started out using INI format for both inventory and secrets. Eventually I needed to use loops and switched the inventory file to YAML format. Now I need to use loops to set the secrets as well, so I want to switch my secrets file from INI to YAML. But ansible-playbook can't seem to read encrypted YAML files.

Here's a toy example of what works, creating a single file for each host. Playbook single_file_test.yaml:

---
- hosts: encrypted_hosts_test
  tasks:
  - name: create file
    ansible.builtin.lineinfile:
      path: "/home/vagrant/{{ file_name }}"
      line: "{{ single_secret }}"
      create: yes

Inventory file hosts.yaml:

all:
  children:
    encrypted_hosts_test:
      hosts:
        machine1:
          file_name: "machine1_file"
        machine2:
          file_name: "machine2_file"

Encrypted secrets file hosts.vault:

[encrypted_hosts_test]
machine1 single_secret="123"
machine2 single_secret="234"

Command:

ansible-playbook --ask-vault-pass -i hosts.yaml -i hosts.vault single_file_test.yaml

Then here's the more complicated playbook I want, multi_file_test.yaml:

---
- hosts: encrypted_hosts_test
  tasks:
  - name: create multiple file tags
    ansible.builtin.lineinfile:
      path: "/home/vagrant/{{ item.key }}"
      line: "{{ item.value.tag }}"
      create: yes
    loop: "{{ file_tags | dict2items }}"
  - name: create multiple file secrets
    ansible.builtin.lineinfile:
      path: "/home/vagrant/{{ item.key }}"
      line: "{{ item.value.secret }}"
      create: yes
    loop: "{{ file_secrets | dict2items }}"

Inventory file hosts.yaml:

all:
  children:
    encrypted_hosts_test:
      hosts:
        machine1:
          file_tags:
            fileA:
              tag: "machine1_fileA_tag"
            fileB:
              tag: "machine1_fileB_tag"
        machine2:
          file_tags:
            fileC:
              tag: "machine1_fileC_tag"
            fileD:
              tag: "machine1_fileD_tag"

Since my secrets are not a fixed set of key/value pairs any more, I need a YAML secrets file. If I use cleartext, cleartext_secret.yaml:

all:
  children:
    encrypted_hosts_test:
      hosts:
        machine1:
          file_secrets:
            fileA:
              secret: "234"
            fileB:
              secret: "345"
        machine2:
          file_secrets:
            fileA:
              secret: "567"
            fileB:
              secret: "789"

Command:

ansible-playbook -i hosts.yaml -i cleartext_secret.yaml multi_file_test.yaml

But if that same secret file is encrypted as hosts.yaml.vault, and I use this command:

ansible-playbook --ask-vault-pass -i hosts.yaml -i hosts.yaml.vault multi_file_test.yaml 

I get this error:

[WARNING]:  * Failed to parse /home/mat/Repository/ansible/pi/test/hosts.yaml.vault with ini plugin: Invalid host pattern 'all:' supplied, ending in ':' is not allowed, this
character is reserved to provide a port.
[WARNING]: Unable to parse /home/mat/Repository/ansible/pi/test/hosts.yaml.vault as an inventory source

I guess it's possible that this just doesn't work. But I find it weird that ansible can auto-detect INI or YAML when the files are unencrypted, but can only handle INI format if the files are encrypted. How can I tell ansible-playbook that this file is in YAML format?

This is on Ubuntu Jammy. Ansible version information:

ansible [core 2.15.8]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/mat/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/mat/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Answer 1

TL;DR

Ansible documentation does not explicitly state that the inventory files could be encrypted. It works with INI-formatted inventories, though, but I did not encounter any official example of that. Even if you try to force usage of YAML inventory plugin, Ansible will just issue a warning that it cannot parse that file as an inventory source.

More practical answer

You're trying to use an inventory file as a source for encrypted variables, but the general practice is to store the variables in group_vars and host_vars directories. It applies to secrets, too.

Moreover, encrypting the whole inventory file will decrease the maintainability of your solution:

• you will need to constantly synchronize the structure of "plain" and "secret" inventories
• any change, even not related to the secrets, will require knowledge of the vault password.

You could either follow the tip from the link above, or simplify the things a bit and store the files with secrets without referencing the encrypted variables in the plain ones - but you (and whoever else maintains the project) will need to know their names or be informed enough on how to find them.

If you want to store all your secrets within one file, you can store it separately or add it to group_vars/all directory, create there a structure that will allow to iterate over the hosts, and adjust your playbook to use that structure. Note that in this case the secrets will be available for all hosts.

If you still want to store the secrets within the inventory file, you can encrypt them independently using ansible-vault encrypt_string YOUR_SECRET, which gives something like that (vault password is 123 here):

!vault |
            $ANSIBLE_VAULT;1.1;AES256
            32396331613636333362363965353939336333643730376530316662336335653739356265373466
            6636383434373761386666333637336333623133303132340a363035323736313166643263336439
            61326636386166666534636432646165643235363531336364623333383139666434373838373465
            3831393033373466640a396665373964643533656665626435613262666664633830363461353066
            3163

This could be inserted as a value of a YAML key:

all:
  children:
    encrypted_hosts_test:
      hosts:
        machine1:
          single_secret: !vault |
            $ANSIBLE_VAULT;1.1;AES256
            32396331613636333362363965353939336333643730376530316662336335653739356265373466
            6636383434373761386666333637336333623133303132340a363035323736313166643263336439
            61326636386166666534636432646165643235363531336364623333383139666434373838373465
            3831393033373466640a396665373964643533656665626435613262666664633830363461353066
            3163