After invoking "gcloud app deploy" for my nodejs app engine project, google cloud build job fails with the following error. This build has been working for several months and it suddenly stopped working.
Step #6 - "exporter": ERROR: failed to export: failed to write image to the following tags: [us.gcr.io/<my-project-id>/app-engine-tmp/app/ttl-2h/default/buildpack-app:latest: GET https://storage.googleapis.com/us.artifacts.<my-project-id>.appspot.com/containers/images/sha256:338cfa338ecd406d9ed339655df9d41c4c203d015b7add07b537b97e2eaa3ca0?access_token=REDACTED: unsupported status code 404; body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.<my-project-id>.appspot.com/containers/images/sha256:338cfa338ecd406d9ed339655df9d41c4c203d015b7add07b537b97e2eaa3ca0</Details></Error>]
Finished Step #6 - "exporter"
ERROR
ERROR: build step 6 "us.gcr.io/gae-runtimes/buildpacks/nodejs10/builder:nodejs10_20200907_10_22_0_RC00" failed: step exited with non-zero status: 246
I tried pulling the image directly from gcr and it failed when attempting to download that layer. I checked the gcp bucket and indeed the docker layer with sha256 338cf... is missing.
Is there a way to force GCP to re-pull that image so that buildpack-app image will be re-downloaded? Perhaps, there is a way to "reset" app engine?
Fixed this issue by deleting the offending container in google container registry. This obviously causes GCP to re-pull the image!