My custom model's loss converged to a high value(Tensorflow)

Question

I'm training my custom model that detects only person.
I used tensorflow object detection API and followed this github document.
I got images from coco dataset.
400 test images and 1600 train images are prepared.
Here is my train config.

train_config: {
  batch_size: 6
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.0001
          decay_steps: 250
          decay_factor: 0.9
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }

And environment.

Tensorflow : 1.13.1 gpu
GPU : GTX 1070
CDUA : 10.0
cuDNN : 7.4.2

According to above github document, the loss should be under 2.
But my model's loss always converges to 4. Is there any problems??
I can't figure out what is wrong...
Thanks for all help.