How do I "merge" my output into an existing YAML file with PyYAML?

Question

I am trying to add data to foo.yaml from a string like this:

foo.yaml

# other stuff in the file
packages: # doesn't have to exist
  - 1
  - 2
  - 3
# other stuff in the file

the string

packages:
  - 4
  - 5

The 2 files aren't always the same formatting. I can't just append the other stuff because the packages: group may already exist and if it does it doesn't have to be at the end of the file.

I have tried just appending to the file but I realized it wouldn't work because packages: might already exist. Also, while I was attempting to append like that the newlines in the string got written to the file as the \n symbol instead of as an actual newline.

Answer 1

You should not be using PyYAML for this. It only support the YAML 1.1 specification, which was outdated in 2009. PyYAML will also happely delete any comments from your YAML files.

Load both files using ruamel.yaml and update the first file using setdefault and extend to cope with packages potentially not being available in the first file:

import sys
import ruamel.yaml
from pathlib import Path

    
yaml = ruamel.yaml.YAML()
yaml.indent(sequence=4, offset=2)
yaml.preserve_quotes = True
foo = yaml.load(Path('foo.yaml'))
extra = yaml.load(Path('extra.yaml'))

# optionally move comment
if 'packages' in foo:
    fpc = foo['packages'].ca
    l = len(foo['packages']) - 1
    if l in fpc.items:
        le = len(extra['packages']) + l
        foo['packages'].ca.items[le] = fpc.items.pop(l)

foo.setdefault('packages', []).extend(extra['packages'])
yaml.dump(foo, sys.stdout)

which gives:

# other stuff in the file
packages: # doesn't have to exist
  - 1
  - 2
  - 3
  - 4
  - 5
# other stuff in the file

The second # other stuf in the file comment gets attached to the last element of the sequence. Extending the list loaded from that sequence would not move the comment, the optonal code there is to move this explicitly to the end of the new sequence.

Disclaimer: I am the author of ruamel.yaml