Add Flink Job Jar in Docker Setup and run Job via Flink Rest API

Question

We're running Flink in Cluster Session mode and automatically add Jars in the Dockerfile:

ADD pipeline-fat.jar /opt/flink/usrlib/pipeline-fat.jar

So that we can run this Jar via the Flink Rest API without the need to upload the Jar in advance:

POST http://localhost:8081/:jarid/run

But the "static" Jar is now shown, to get the :jarid:

GET http://localhost:8081/jars

---

So my question is:

Is it possible to run a userlib jar using the Flink Rest API?

Or can you only reference such jars via

• CLI flink run -d -c ${JOB_CLASS_NAME} /job.jar
• and standalone-job --job-classname com.job.ClassName Mode?

---

My alternative approach (workaround) would be to upload the jar in the Docker entrypoint.sh of the jobmanager container:

curl -X POST http://localhost:8084/jars/upload \
    -H "Expect:" \
    -F "jarfile=@./pipeline-fat.jar"

Answer 1

I managed to run a userlib jar using the command line interface.

I edited docker compose to run custom docker-entrypoint.sh. I've add to original docker-entrypoint.sh

run_user_jars() {
echo "Starting user jars"
exec ./bin/flink run /opt/flink/usrlib/my-job-0.1.jar & }

run_user_jars
...

And edit original entrypoint for jobmanager in docker-compose.yml file

entrypoint: ["bash", "/opt/flink/usrlib/custom-docker-entrypoint.sh"]

Answer 2

I believe that it is unfortunately not possible to currently start a flink cluster in session mode with a jar pre-baked in the docker image, and then start the job using the REST API commands (as you showed).

However your workaround approach seems like a good idea to me. I would be curious to see if it worked for you in practice.