Using for loop and other declared variables in ec2 userdata cloudformation

Question

I am new to cloudformation yaml.
I have the following which I need to use as userdata in launch configuration:

  UserData:
    Fn::Base64: !Sub 
      - |
        #!/bin/bash -x
        apt-get -y update && apt install -y awscli mysql-client libmysqlclient-dev python-pip
          for item in certs user_prop config log
          do
            echo "... preparing ${item} database and config"
            MYSQL_DB_NAME="ext_as_${item}"
            LOCAL_DB_NAME=$(echo ${item}|tr -d '_')
            LOCAL_DB_FILE="/path/to/db/${LOCAL_DB_NAME}"
            DB_KEY="${item}_db"
            #- set db configuration value
            sed -i "s|${DB_KEY}=.*|${DB_KEY}==mysql://${DB_FQDN}/${MYSQL_DB_NAME}|" /path/to/config.conf
            #- create mysql db
            mysql --defaults-file=${MYSQL_PREF} -e "CREATE DATABASE IF NOT EXISTS $${MYSQL_DB_NAME};"
            #- import local DB schema into MySql if no tables exist
            mysql --defaults-file=$${MYSQL_PREF} --silent --skip-column-names \
            -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = '${MYSQL_DB_NAME}';"|grep -e ^0 -q \
            && ./dbcvt -t ${item} -s sqlite:///${LOCAL_DB_FILE} -d mysql://${DB_FQDN}/${MYSQL_DB_NAME} -p ${MYSQL_PREF}
          done
        popd
      - 
        DB_FQDN:
          'Fn::ImportValue': 'OVPNConfigDbEndpoint'

Now the problem is DB_FQDN gets resolved but LOCAL_DB_FILE, LOCAL_DB_NAME, DB_KEY, MYSQL_DB_NAME, and MYSQL_PREF don't get resolved. Now the questions are:

1. If I hard-code the other values, how do I use a for loop in a userdata shell script ?
2. Do I have to always use a Fn::Join as mentioned in https://stackoverflow.com/a/54535452/4035062 by Alex ?
   using Fn::Join to write a simple shell script seems quite complicated.
3. What does + mean in !Sub |+ ?

Answer 1

The problem you have there is that you want the variables like ${item} to be expanded inside the Bash Shell whereas the Fn::Sub Cloudformation intrinsic function also interprets ${item} as a variable to expand.

As mentioned in the docs:

To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.

Keep in mind, however, that the braces in Bash are usually optional. That is to say, you can usually rewrite ${item} simply as $item.

A quick look at your code suggests to me that you can rewrite it like this and it should be fine:

UserData:
  Fn::Base64: !Sub 
   - |
     #!/bin/bash -x
     apt-get -y update && apt install -y awscli mysql-client libmysqlclient-dev python-pip
     for item in certs user_prop config log
     do
       echo "... preparing $item database and config"
       MYSQL_DB_NAME="ext_as_$item"
       LOCAL_DB_NAME=$(echo $item | tr -d '_')
       LOCAL_DB_FILE="/path/to/db/$LOCAL_DB_NAME"
       DB_KEY="${!item}_db"
       #- set db configuration value
       sed -i "s|$DB_KEY=.*|$DB_KEY==mysql://${DB_FQDN}/$MYSQL_DB_NAME|" /path/to/config.conf
       #- create mysql db
       mysql --defaults-file=$MYSQL_PREF -e "CREATE DATABASE IF NOT EXISTS $MYSQL_DB_NAME;"
       #- import local DB schema into MySql if no tables exist
       mysql --defaults-file=$MYSQL_PREF --silent --skip-column-names \
       -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = '$MYSQL_DB_NAME';"|grep -e ^0 -q \
       && ./dbcvt -t $item -s sqlite:///$LOCAL_DB_FILE -d mysql://${DB_FQDN}/$MYSQL_DB_NAME -p $MYSQL_PREF
     done
     popd
   - 
     DB_FQDN:
       'Fn::ImportValue': 'OVPNConfigDbEndpoint'

Points to note well:

• See that on the DB_KEY the ${!item} notation is required, because Bash would otherwise expand the variable ${item_db} without the braces there.

• In the case of ${DB_FQDN} you of course need to use ${DB_FQDN} because in that case you do want the substitution by Fb::Sub.

• Note also that I fixed some typos in your Bash. You had $$ as a typo in 2 places.

To your other questions:

• No you don't need to use Fn::Join. That question you were looking at was about if someone for some reason did want to interpolate Fn::Join. That answer shows one way you can do that.

• The notation |+ is a feature of the YAML language. It just says to keep the newlines after the block. See this YAML cheatsheet.