Python Subprocess call to invoke java jar files with JAVA_OPTS

Question

I am trying to make a subprocess call from my python script to a java class built inside a jar. I am running the python code on a docker container in AWS Batch. I set the CLASSPATH environment variable in the Dockerfile to include the directory containing the jar file.

ENV CLASSPATH /path/to/dir/containing/jar/file

When I pass the entire command with arguments as a string to subprocess, it works fine.

runnable_command = "java $JAVA_OPTS " \
                   "RunCommand " \
                   "-b arg_b"

sp = subprocess.Popen(runnable_command,
                      stdin=subprocess.PIPE,
                      stdout=subprocess.PIPE,
                      stderr=subprocess.STDOUT,
                      close_fds=True,
                      shell=True,
                      universal_newlines=True,
                      env=os.environ)

result, stderr_data = sp.communicate()
print(result)

But for that I had to make the variable "shell=True" which has a security risk. So I modified the variable 'shell=False" and I pass in the command and arguments as a list to the subprocess. This also works fine.

runnable_command = ["/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.x86_64/jre/bin/java", "$JAVA_OPTS", "<ClassName>"]

However, I am setting JAVA_OPTS environment variable in the Dockerfile to pass the log4j configuration file to JVM.

ENV JAVA_OPTS="-Dlog4j.configurationFile=/opt/amazon/lib/log4j2.xml"

This is important because I want to pipe the logs from this java script to my python script. When I add JAVA_OPTS to the command, it fails with the following error:

runnable_command = ["/usr/lib/jvm/java-1.8.0-jdk-1.8.0.252.b09-2.x86_64/jre/bin/java", "$JAVA_OPTS", "<ClassName>"]

I am not able to pass JAVA_OPTS to the list of args in the subprocess command. It fails to find the log4j.xml file. I followed this question from stackoverflow but it fails with the same error, even after adding the JAVA_OPTS to the "env" argument.

'Error: Could not find or load main class $JAVA_OPTS\n'

Also, when I pass the arguments as a list, I am not able to run 'java' but I am forced to pass the absolute path of java executable.

Can someone help me with the following questions?

1. How can i pass the log4j configuration to this java command?
2. Why am I having to pass the absolute path to java command when running subprocess with a list and not a string?