docker buildx build automatically pulls base image

Question

I recently updated a Docker workflow to start using buildx. I quickly noticed that the build of the Dockerfile was often restarting from scratch, even if from the last build I only added a layer "at the end". Upon further inspection, It seems like the build is always trying to pull the base images (e.g. FROM ubuntu:22.04) and so, this results in invalidating the whole cache.

Before moving to buildx I was able to control this behavior by passing (or not) the --pull argument, and by looking at the new documentation this argument is still supported. Then why am I seeing this behavior?

At a high level, this is what I'm doing.

docker buildx create --name my-builder

docker build \
    --builder my-builder \
    --platform=linux/amd64,linux/arm64
    --push
    --tag my-tag \
    --file path/to/Dockerfile \
    .

Thank you