How do I build a YAML list using yq?

Question

I am trying to use Mike Farrah's yq utility to build YAML files from within a bash script. Below is an example of the template I'm using:

---
name: <name>
description: <description>
type: <type>
labels: [label]
data:
  - value: <value>
    operation: <operation>
    labelValues: [value]

The labels and labelValues fields are a list of strings. For example:

labels: [string1, string2, ...]

or

labels:
  - string1
  - string2
  - ...

To add some further complexity, the data field is a list of objects, consisting of the value, operation and labelValues fields.

I'm trying to figure out how to build the labels list, using a bash script array of values. Not having much luck. The closest I've gotten is with the following code:

The array:

$ echo ${labels[@]}
project group

The command:

for lbl in ${labels[@]}; do
  yq '.labels.[] |= "'$lbl'"' metric-spec-template.yaml
done

This is what I need:

---
name: <name>
description: <description>
type: <type>
labels: [project, group]
data:
  - value: <value>
    operation: <operation>
    labelValues: <labelValues>

This is what I get:

name: <name>
description: <description>
type: <type>
labels: [project]
data:
  - value: <value>
    operation: <operation>
    labelValues: <labelValues>
name: <name>
description: <description>
type: <type>
labels: [group]
data:
  - value: <value>
    operation: <operation>
    labelValues: <labelValues>

The user guide is nicely written, but I can't figure out how to make this work as needed. I haven't even begun to think about how I would build the list of data items.

Are there any comprehensive tutorials on using yq? I haven't been able to find much beyond the main user guide.

Answer 1

Your approach fails because in the for loop you're calling yq separately for each item, but such a call never reads as input the results of the previous iteration, so there's nothing yq could "update". You should save the intermediary document in a shell variable, or save it to a temporary file, or directly update the source file itself (the latter two make use of the --inplace (or -i) flag).

# using a shell variable

declare -a labels=(project group)
doc="$(yq '.labels = []' template.yaml)" # resets array, stores in var
for lbl in "${labels[@]}"; do
  doc="$(lbl="$lbl" yq '.labels += strenv(lbl)' <<< "$doc")"
done
printf '%s\n' "$doc" # outputs result

# using a temporary file

declare -a labels=(project group)
tmp="$(mktemp)" # creates a temporary file
yq '.labels = []' template.yaml > "$tmp" # resets array, saves to temp file
for lbl in "${labels[@]}"; do
  lbl="$lbl" yq -i '.labels += strenv(lbl)' "$tmp" # modifies temp file
done
cat "$tmp"; rm -f "$tmp" # outputs result, removes temp file

# using the source file

declare -a labels=(project group)
yq -i '.labels = []' template.yaml # resets array, modifies src file
for lbl in "${labels[@]}"; do
  lbl="$lbl" yq -i '.labels += strenv(lbl)' template.yaml # modifies src file
done
cat template.yaml # outputs result from src file

Ideally, though, you should move the entire loop into yq, and provide to it all the data necessary for the iteration at once, so you'd end up with just one call to yq which should then also perform better.

Unfortunately, you cannot just import a bash array as a YAML array, as yq imports variables through the environment, and "There isn't really a good way to encode an array variable into the environment" (by Chet Ramey, the current maintainer of bash).

Your remaining options would be:

• prepending the items with dashes, each on its separate line, so the resulting string can be conventionally imported using the environment, and interpreted by yq as another YAML input comprising a list of strings (Note that this is an ad-hoc improvident hit-or-miss YAML encoding which would likely break if an item contained certain special characters (e.g. line breaks) capable to invalidate the intended encoding.)
• converting the array just into a string of space-separated items, which could then be imported as such, and split back into an array by yq (Note that this still relies on the items not containing whitespace characters, but as you've used echo ${labels[@]} and for lbl in ${labels[@]}; do yourself without quoting the variables, it's fair to assume that they don't.)
• using another YAML processor that provides other means for data imports, like kislyuk/yq or itchyny/gojq, for example. The former converts between YAML and JSON, and then uses jq internally for the JSON processing, while the latter is a full rewrite of jq that additionally implements a YAML processor. Thus, both can make use of jq's --args flag which allows for the provision of strings as arguments on the command line, which can then be accessed using the built-in $ARGS variable.

# using an ad-hoc YAML encoding

declare -a labels=(project group)
lbls="$(printf -- '- %s\n' "${labels[@]}")" yq '.labels = env(lbls)' template.yaml

# converting into a space-separated string

declare -a labels=(project group)
lbls="${labels[*]}" yq '.labels = strenv(lbls) / " "' template.yaml

# using another YAML processor, here kislyuk/yq or itchyny/gojq
# (aliasing is optional, it just illustrates that both use the same jq filter)

declare -a labels=(project group)
alias yq2='yq -y' # if using kislyuk/yq
alias yq2='gojq --yaml-input --yaml-output' # if using itchyny/gojq
yq2 '.labels = $ARGS.positional' template.yaml --args "${labels[@]}"