Company RefTeX slow when searching for labels

Question

I am writing on a large LaTeX Document using Emacs with AUCTeX, RefTeX and Company for autocompletion. RefTeX for citations is working fine and finds my bibliography entries really fast. However, when RefTeX searches for labels it has to scan every file used in the project, this takes several seconds and is done everytime I move the point in a \ref{} command. Also while scanning for files, I cannot make additional keyboard input, making me wait until scanning is finished everytime.
Here is the relevant part of my Emacs configuration:

;;------------------------------------------------------------------------------
;; Company -> Autocomplete
;;------------------------------------------------------------------------------

(use-package company
  :config
  (setq company-idle-delay 0
        company-minimum-prefix-length 4
        company-selection-wrap-around t)
  :bind
  (("C-<tab>" . company-complete))
  )

(global-company-mode t)

(use-package company-math :ensure t)
(use-package company-reftex :ensure t)
(use-package company-bibtex :ensure t)

(add-hook 'LaTeX-mode-hook
        (lambda()
        (make-local-variable 'company-backends)
        (setq company-backends '((company-files company-yasnippet)
                               (company-reftex-citations
                                company-reftex-labels
                                company-bibtex
                                company-auctex-macros
                                company-auctex-symbols
                                company-auctex-environments
                                company-keywords
                                company-latex-commands
                                company-auctex-environments
                                company-math-symbols-latex
                                company-ipa-symbols-unicode)
                                company-dabbrev))))

;;------------------------------------------------------------------------------
;; LaTeX
;;------------------------------------------------------------------------------

(setq-default TeX-output-dir "build")
(setq-default TeX-master nil)

;; RefTex
(add-hook 'LaTeX-mode-hook 'turn-on-reftex) ; Enable RefTex in AUCTex Latex mode
(add-hook 'latex-mode-hook 'turn-on-reftex) ; Enable RefTex in Emacs Latex mode

;; Enable LaTeX Preview Pane
;(add-hook 'LaTeX-mode-hook 'latex-preview-pane-mode)
;(add-hook 'latex-mode-hook 'latex-preview-pane-mode)

(setq-default reftex-plug-into-AUCTeX t)
(setq-default reftex-default-bibliography '("./bibliography/Masterthesis.bib"))
(setq-default reftex-bibliography-commands '("bibliography" "nobibliography" "addbibresource"))
(setq reftex-use-external-file-finders t)
(setq reftex-external-file-finders
      '(("tex". "kpsewhich -format=.tex %f")
                ("bib". "kpsewhich -format=.bib %f")))

Can I speed up the RefTeX process for scanning labels? Or is this there at least a way to disable the automatic scanning?

I am new to emacs and think my problem could result from a faulty configuration. Thanks for your help!