Jenkins multi-branch pipeling stuck on an old Jenkinsfile

Question

I am practicing a CI workflow using cowsay as an app with a local git repository. I have a docker-compose that builds three docker containers: a Jenkins, a gitLab and an Artifactory. all three use a named volume to store consistent data. I am running a multi-branch pipeline based on a Jenkinsfile (declarative) and a webhook for push events. So far so good. BUT - Jenkins is STUCK on an old version of Jenkinsfile. Solutions I tried:

1. I manually emptied jenkins-home from all versions of Jenkinsfile I could find (I looked in ~/jobs/<job_name> and in ~/caches )
2. I deleted Jenkins entirely and re-built the image, including a volume prune.
3. I migrated files manually into a new repository
4. I used "git filter-branch" to delete all version of Jenkinsfile from all branches and commits.
5. I manually rewrote (not copy-paste) the Jenkinsfile
6. I event went as desprate as to clean my entire browser cache for no reason.

Jenknis refuses to re-cache and in some bizzare way keeps running this line: "docker build -t cowsay/master ." which is non-existant in the Jenkinsfile.

I can't find any more caches to clean other than. Help?

EDIT: I am not running any commands. This is configured so the Jenkins job runs upon a 'git push' to any branch. The error I am seeing is this line: +docker build -t 'cowsay' . It's a line that used to be in the Jenkinsfile in one of the stages, but isn't for at least the last 5 commits/pushes. This line causes an error that fails the job.

Answer 1

I have never experienced this failure myself but here a couple of things I would try to debug what's wrong:

1. I would first ensure that the is configured to use a Jenkinsfile from git, and not a static script. To do that, go to your multibranch pipeline, click "Configure", and in the "Build Configuration" block it should say that the mode is "by Jenkinsfile" and the path is the relative path of your Jenkinsfile in the repo. If you go to the per branch build jobs, and you click on "View Configuration", you should see that their pipeline definition is "Pipeline from multibranch configuration".
2. I would verify that the issue you're seeing has to do with Jenkinsfile caching and not with one of your scripts running a command you didn't expect (eg if you run a shell script or a make file, they might be calling the command that's failing). To do that, I'd add an echo at the top of the Jenkinsfile, build the job and see if the echo is in the job output. If it is, you're getting latest from Jenkins and something else is calling that command you removed from your Jenkinsfile.
3. If you don't see the echo command mentioned in step 2 you are definitely not getting latest. You mentioned you use a local git repo. How did you configure the git scm? Is it possible that jenkins is waiting for push notifications from a remote server that you're not pushing your local changes to? If you're seeing builds getting automatically triggers on every commit you make, that would probably eliminate this case.