I am new to skaffold, k8s, docker set and I've been having trouble building my application on a cluster locally.
I have a code repository that is trying to pull a private NPM package but when building it loses the .npmrc file or the npm secret.
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@sh1ba%2fcommon - Not found
npm ERR! 404
npm ERR! 404 '@sh1ba/common@^1.0.3' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-06-02T06_08_57_246Z-debug.log
unable to stream build output: The command '/bin/sh -c npm install' returned a non-zero code: 1. Please fix the Dockerfile and try again..
Ideally I'd like to avoid hard coding the secret into the file and use a k8s environment variable to pass in the key to docker as a secret. I am able to (kind of) do it with the docker build command:
- with "--build-args" with npm secret (the not safe way)
- with "--secret" with npm secret (the better way)
- copying the .npmrc file directly,
npm install
ing and deleting it right after
The issue arises when I try to build it using kubernetes/skaffold. After running, it doesn't seem like any of the args, env variables, or even the .npmrc file is found. When checking in the dockerfile for clues I was able to identify that nothing was being passed over from the manifest (args defined, .npmrc file, etc) to the dockerfile.
Below is the manifest for the application:
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: auth
env:
- name: NPM_SECRET
valueFrom:
secretKeyRef:
name: npm-secret
key: NPM_SECRET
args: ["--no-cache", "--progress=plain", "--secret", "id=npmrc,src=.npmrc"]
Here's the code in the dockerfile:
# syntax=docker/dockerfile:1.2
# --------------> The build image
FROM node:alpine AS build
WORKDIR /app
COPY package*.json .
RUN --mount=type=secret,mode=0644,id=npmrc,target=/app/.npmrc \
npm install
# --------------> The production image
FROM node:alpine
WORKDIR /app
COPY package.json .
COPY tsconfig.json .
COPY src .
COPY prisma .
COPY --chown=node:node --from=build /app/node_modules /app/node_modules
COPY --chown=node:node . /app
s
RUN npm run build
CMD ["npm", "start"]
And also the skaffold file:
apiVersion: skaffold/v2alpha3
kind: Config
deploy:
kubectl:
manifests:
- ./infra/k8s/*
- ./infra/k8s-dev/*
build:
local:
push: false
artifacts:
- image: auth
context: auth
docker:
dockerfile: Dockerfile
sync:
manual:
- src: 'src/**/*.ts'
dest: .
A few notes:
- I can't locate the .npmrc file regardless of where I copy and paste it (in auth, in the manifest, in skaffold and in the ~/ directories)
- I would like to also make it semi-usable (pretty reusable) in production too so that I don't need to do a complete overhaul if possible (but if this is bad practice I'd like to hear more about it)
- I've been able to make it work with buildArgs in the skaffold.yaml file but I'm not sure how that would translate into a production environment as I can't pass build args from kubernetes to docker (and I read that it isn't safe and that secrets should be used)
- The args in the manifest are throwing this error too (server runs if the args are commented out):
- deployment/auth-depl: container auth terminated with exit code 9
- pod/auth-depl-85fb8975d8-4rh9r: container auth terminated with exit code 9
> [auth-depl-85fb8975d8-4rh9r auth] node: bad option: --progress=plain
> [auth-depl-85fb8975d8-4rh9r auth] node: bad option: --secret
- deployment/auth-depl failed. Error: container auth terminated with exit code 9.
Any insight would be amazing, I've been fiddling with this for far too long now.
Thank you!
Building and deploying an image to Kubernetes is at three levels:
Docker is not involved in #3. (This is only partially true, since some clusters use Docker to run the containers too, but that's a hidden detail and is also changing.)
There are two places where you might communicate secrets:
--build-args
or mounting secrets with--secret
(both require Buildkit)Skaffold supports passing build-time secrets, like your npm password, with Docker's
--build-args
and--secret
flags, though they are slightly renamed.buildArgs
supports Go-style templating, so you can reference environment variables likeMYSECRET
as{{.MYSECRET}}
:Then you can reference
MYSECRET
within yourDockerfile
:Note that build-args are not propagated into your container unless you explicitly assign it via an
ENV MYSECRET=${MYSECRET}
.If the secret is in a local file, you can use the
secret
field in theskaffold.yaml
:and you'd then reference the secret as you are in your
Dockerfile
:Now in your
Deployment
, you're attempting to settingargs
for your container:The
args
field overrides theCMD
directive set in your image. This field is used to provide command-line arguments provided to your image's entrypoint, which is likelynode
. If you want to reference a secret in a running container on a cluster, you'd use aSecret
orConfigMap
.