Docker ignore file for multistage build

Question

I need to exclude file in Docker COPY for the first stage but use it in another COPY for the second stage, because it breaks docker layer caching.

My folder structure looks like:

src/
public/
.env
package.json
nginx.conf

My Dockerfile looks like this:

FROM node:14 as builder                   # line-1
WORKDIR /app                              # line-2
COPY . /app                               # line-3 nginx.conf is copied but not needed
RUN yarn && yarn build                    # line-4

FROM nginx:1.21.1                         # line-5
WORKDIR /app                              # line-6
COPY --from=builder /app/build /app/build # line-7
COPY nginx.conf /app                      # line-8 nginx.conf is copied

CMD nginx -c /app/nginx.conf              # line-9

When I change nginx.conf docker layer caching becomes invalidated because of the 3rd line.

I cannot use .dockerignore because it will ignore it for both stages.

Obvious solution is to copy all the needed folders one by one, but that seems to be unreliable because project structure can change from time to time and I'd like to avoid extra effort of Dockerfile maintenance & updating.

Is there shorter way to overcome this apart from that is mentioned here?