Received error in GitLab: "The repository for this project does not exist."

Question

I run the GitLab from docker-compose. I had repositories with content and everything had been working as expected but now it says "The repository for this project does not exist." and I cannot get back my existing repositories and the content of them.

In /etc/gitlab/gitlab.rb file the git_data_dirs points to /var/opt/gitlab/git-data as default. The user/group set to git for /var/opt/gitlab/git-data/repositories recursively.

Content of /var/opt/gitlab/git-data/repositories:

ll /var/opt/gitlab/git-data/repositories
total 84
drwxrws---  3 git git  4096 Dec 22 01:32 +gitaly/
drwxrws---  4 git git  4096 Dec 22 09:28 ./
drwx------  3 git git  4096 Aug 31  2020 ../
-rwxrwx---  1 git git    64 Dec 22 09:28 .gitaly-metadata*
-rwxrwx---  1 git git    80 Dec 22 03:15 .gitaly-metadata.locked*
drwxrws--- 16 git git  4096 Apr 19  2021 @hashed/
-rwxrwx---  1 git git 57607 Dec 22 03:15 __$$RECOVERY_README$$__.html

Content of /var/opt/gitlab/git-data/repositories/@hashed: (It seems my Git repos are available)

ll /var/opt/gitlab/git-data/repositories/@hashed
total 64
drwxrws--- 16 git git 4096 Apr 19  2021 ./
drwxrws---  4 git git 4096 Dec 22 09:28 ../
drwxrws---  3 git git 4096 Oct 21  2020 19/
drwxrws---  3 git git 4096 Oct 21  2020 2c/
drwxrws---  3 git git 4096 Dec  4  2020 3f/
drwxrws---  3 git git 4096 Oct 28  2020 4a/
drwxrws---  3 git git 4096 Sep 20  2020 4b/
drwxrws---  3 git git 4096 Sep 20  2020 4e/
drwxrws---  3 git git 4096 Nov  5  2020 4f/
drwxrws---  4 git git 4096 Nov 11  2020 6b/
drwxrws---  3 git git 4096 Oct 21  2020 79/
drwxrws---  3 git git 4096 Dec 11  2020 85/
drwxrws---  3 git git 4096 Aug 28  2020 d4/
drwxrws---  3 git git 4096 Apr 19  2021 e6/
drwxrws---  3 git git 4096 Oct 12  2020 e7/
drwxrws---  3 git git 4096 Sep 20  2020 ef/

The repos are even not accessible from my local PC:

>>> git fetch
Username for 'https://my_host.com': milan.balazs
Password for 'https://milan.balazs@my_host.com': 
remote: A repository for this project does not exist yet.
fatal: repository 'https://my_host.com/systemdevelopers/tools.git/' not found

I have search and tried many "solution"/work-around but I couldn't solve it.

Tried:

• gitlab-rake cache:clear
• gitlab-rake gitlab:check
• gitlab-ctl reconfigure
• Restart Docker Service
• Restart Docker Swarm
• Restart Docker Daemon

The given error on UI:

My docker-compose.yml:

version: "3.8"

services:
  gitlab:
    image: ${ENVIRONMENT_HOST}:5000/environment/gitlab 
    ports:
      - "80:80"
      - "443:443"
      - "60000:22"
    hostname: "${ENVIRONMENT_HOST}"
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://${ENVIRONMENT_HOST}'
        gitlab_rails['gitlab_shell_ssh_port'] = 60000
        nginx['redirect_http_to_https'] = true
        nginx['ssl_certificate'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/fullchain.pem'
        nginx['ssl_certificate_key'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/privkey.pem'
        gitlab_shell['custom_hooks_dir'] = '/home/git/gitlab-shell/hooks'

    volumes:
      - gitlab_config:/etc/gitlab
      - gitlab_logs:/var/log/gitlab
      - gitlab_data:/var/opt/gitlab
      - /etc/letsencrypt:/etc/letsencrypt
      - ./gitlab/server_hooks/:/home/git/gitlab-shell/hooks
    networks:
      - gitlab_ci_net

  gitlab-runner-1:
    image: gitlab/gitlab-runner:alpine-v13.6.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - gitlab_runner_config_1:/etc/gitlab-runner
    networks:
      - gitlab_ci_net
      - image_registry_net

  gitlab-runner-2:
    image: gitlab/gitlab-runner:alpine-v13.6.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - gitlab_runner_config_2:/etc/gitlab-runner
    networks:
      - gitlab_ci_net
      - image_registry_net

  gitlab-runner-3:
    image: gitlab/gitlab-runner:alpine-v13.6.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - gitlab_runner_config_3:/etc/gitlab-runner
    networks:
      - gitlab_ci_net
      - image_registry_net
volumes:
  gitlab_config: {}
  gitlab_logs: {}
  gitlab_data: {}
  gitlab_runner_config_1: {}
  gitlab_runner_config_2: {}
  gitlab_runner_config_3: {}

networks:
  gitlab_ci_net: {}
  image_registry_net: {}

The ${ENVIRONMENT_HOST}:5000/environment/gitlab Dockerfile:

FROM gitlab/gitlab-ce:13.6.3-ce.0

# Install python3 for server hooks
RUN apt-get update && apt-get install -y python3

# Copy server hooks
COPY --chown=git:git ./server_hooks /home/git/gitlab-shell/hooks

# Give permission to server hooks
RUN find . -name *.py -exec chmod +x {} \

The gitlab-ctl reconfigure command has been ran successfully.

Some strange log parts:

I don't know the following error messages from log folder are related or not but I share them, perhaps they can help in debugging. The used command to find messages root@env:/var# grep -rnw 'log/' -e 'error'

In my understanding the gitaly is not able to recognize the Git repository (I can see this message for more Git repos).

log/gitlab/gitaly/@4000000061c31f1028b9701c.u:437:
{
   "correlation_id":"EyMTcQIIRP9",
   "error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git'",
   "grpc.code":"NotFound",
   "grpc.meta.auth_version":"v2",
   "grpc.meta.client_name":"gitlab-web",
   "grpc.meta.deadline_type":"regular",
   "grpc.method":"FindDefaultBranchName",
   "grpc.request.deadline":"2021-12-22T12:32:41Z",
   "grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName",
   "grpc.request.glProjectPath":"systemdevelopers/database",
   "grpc.request.glRepository":"project-6",
   "grpc.request.repoPath":"@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git",
   "grpc.request.repoStorage":"default",
   "grpc.request.topLevelGroup":"@hashed",
   "grpc.service":"gitaly.RefService",
   "grpc.start_time":"2021-12-22T12:32:31Z",
   "grpc.time_ms":0.252,
   "level":"info",
   "msg":"finished unary call with code NotFound",
   "peer.address":"@",
   "pid":266,
   "span.kind":"server",
   "system":"grpc",
   "time":"2021-12-22T12:32:31.361Z"
}

Other strange message (I see more times in my logs):

log/gitlab/gitlab-rails/application_json.log:42:
{
   "severity":"ERROR",
   "time":"2021-12-22T12:07:41.806Z",
   "correlation_id":null,
   "message":"  \n!! RUNTIME IDENTIFICATION FAILED: Failed to identify runtime for process 52717 (bin/rails)\n  Runtime based configuration settings may not work properly.\n  If you continue to see this error, please file an issue via\n  https://gitlab.com/gitlab-org/gitlab/issues/new\n"
}

Some errors from gitlab-workhorse:

log/gitlab/gitlab-workhorse/@4000000061c31f28343253dc.u:11373:
{
   "command":[
      "exiftool",
      "-all=",
      "--IPTC:all",
      "--XMP-iptcExt:all",
      "-tagsFromFile",
      "@",
      "-ResolutionUnit",
      "-XResolution",
      "-YResolution",
      "-YCbCrSubSampling",
      "-YCbCrPositioning",
      "-BitsPerSample",
      "-ImageHeight",
      "-ImageWidth",
      "-ImageSize",
      "-Copyright",
      "-CopyrightNotice",
      "-Orientation",
      "-"
   ],
   "correlation_id":"ODWhqqnLNH2",
   "error":"exit status 1",
   "level":"info",
   "msg":"exiftool command failed",
   "stderr":"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0Error: Writing of this type of file is not supported - -\n",
   "time":"2021-12-22T12:44:50Z"
}


log/gitlab/gitlab-workhorse/@4000000061c31f28343253dc.u:11374:
{
   "correlation_id":"ODWhqqnLNH2",
   "error":"error while removing EXIF",
   "level":"error",
   "method":"POST",
   "msg":"error",
   "time":"2021-12-22T12:44:50Z",
   "uri":"/uploads/user"
}

log/gitlab/gitlab-workhorse/@4000000061c343b71091c0ac.u:4:
{
   "correlation_id":"OKWMhVphvr5",
   "duration_ms":0,
   "error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused",
   "level":"error",
   "method":"GET",
   "msg":"error",
   "time":"2021-12-22T12:51:32Z",
   "uri":"/help"
}

Results of GitLab environment info:

root@env:/# gitlab-rake gitlab:env:info

System information
System:     
Current User:   git
Using RVM:  no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.1
Redis Version:  5.0.9
Git Version:    2.29.0
Sidekiq Version:5.2.9
Go Version: unknown

GitLab information
Version:    13.6.3
Revision:   857c6c6a6a9
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.9
URL:        https://my_host.com
HTTP Clone URL: https://my_host.com/some-group/some-project.git
SSH Clone URL:  ssh://git@my_host.com:60000/some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:    13.13.0
Repository storage paths:
- default:  /storage/data/gitlab/git-data/repositories
GitLab Shell path:      /opt/gitlab/embedded/service/gitlab-shell
Git:        /opt/gitlab/embedded/bin/git

Results of GitLab application Check:

root@env:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
11/1 ... yes
5/2 ... yes
11/3 ... yes
11/4 ... yes
11/5 ... yes
11/6 ... yes
11/7 ... yes
11/8 ... yes
11/9 ... yes
11/10 ... yes
11/11 ... yes
11/12 ... yes
11/13 ... yes
11/14 ... yes
11/15 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 14
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

I have already written a bug ticket for GitLab but I haven't got any feedback yet: https://gitlab.com/gitlab-org/gitlab/-/issues/349149

I would be very grateful if somebody could provide some hits how I can recover my system.