Nvidia digits on TX2 Error code 1

Question

I am new to Digits and TX2. I am trying to create object detection model using the tutorial from: https://github.com/dusty-nv/jetson-inference

I created dataset sucessfully. The issue is with the model

While creating a model, I am getting the following error.

    Memory required for data: 3268934784
    creating layer bbox_loss        
    Creating Layer bbox_loss
    bbox_loss <- bboxes-obj-masked-norm
    bbox_loss <- bbox-obj-label-norm
    bbox_loss -> loss_bbox
    Setting up bbox_loss
    Top shape: (1)
    with loss weight 2
    Memory required for data: 3268934788
    Creating layer coverage_loss
    Creating Layer coverage_loss
    coverage_loss <- coverage_coverage/sig_0_split_0
    coverage_loss <- coverage-label_slice-label_4_split_0
    coverage_loss -> loss_coverage
    Setting up coverage_loss
    Top shape: (1)
    with loss weight 1
    Memory required for data: 3268934792
    Creating layer cluster

The job directory information on the left is:

Job Directory
/home/nvidia/DIGITS/digits/jobs/20180816-161051-e67a
Disk Size
0 B
Network (train/val)
train_val.prototxt
Network (deploy)
deploy.prototxt
Network (original)
original.prototxt
Solver
solver.prototxt
Raw caffe output
caffe_output.log
Pretrained Model
/home/nvidia/bvlc_googlenet.caffemodel.4
Visualizations
Tensorboard

The error on the server is

2018-08-16 16:10:53 [20180816-161051-e67a] [INFO ] Task subprocess args: "/home/nvidia/Caffe/caffe/build/tools/caffe train --solver=/home/nvidia/DIGITS/digits/jobs/20180816-161051-e67a/solver.prototxt --gpu=0 --weights=/home/nvidia/bvlc_googlenet.caffemodel.4"
2018-08-16 16:11:00 [20180816-161051-e67a] [ERROR] Train Caffe Model task failed with error code 1

I have no idea on how to free up memory as I have more than 2 gb available in the job directory. Please help me. Thanks in advance.

Answer 1

Had the same issue for the last few days, maybe it will help someone in the future. Firstly, make sure that you have the right version of protobuf. You can check it with:

protoc --version 

If it's 2.* you have to update to 3.*, for example to build it as listed here https://github.com/NVIDIA/DIGITS/blob/digits-6.0/docs/BuildProtobuf.md, and then rebuild the Caffe. Also, make sure that you have the compatible version of pip package of protobuf. For me the following version is working well right now for Digits and Caffe from the tutorial https://github.com/dusty-nv/jetson-inference :

pip install --user --upgrade protobuf==3.1.0.post1