(helmet) c:\Users\square\Tensorflow\models\research>python C:/Users/square/Tensorflow/models/research/object_detection/model_main_tf2.py --pipeline_config_path=C:/Users/square/Tensorflow/models/research/object_detection/configs/tf2/ssd_mobilenet_v2_320x320_coco17_tpu-8.config --model_dir=D:/training_output --num_train_steps=50000 --sample_1_of_n_eval_examples=1 --alsologtostderr
2023-10-07 17:00:31.341113: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2023-10-07 17:00:31.341848: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow_addons\utils\tfa_eol_msg.py:23: UserWarning:
TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.
Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP).
For more information see: https://github.com/tensorflow/addons/issues/2807
warnings.warn(
C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow_addons\utils\ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.11.0 and strictly below 2.14.0 (nightly versions are not supported).
The versions of TensorFlow you are currently using is 2.10.1 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
warnings.warn(
2023-10-07 17:00:50.556325: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2023-10-07 17:00:50.557837: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2023-10-07 17:00:50.559270: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2023-10-07 17:00:50.560725: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2023-10-07 17:00:50.562170: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2023-10-07 17:00:50.563661: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
2023-10-07 17:00:50.565100: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2023-10-07 17:00:50.566552: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2023-10-07 17:00:50.566706: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2023-10-07 17:00:50.582724: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
W1007 17:00:50.598761 6532 cross_device_ops.py:1387] There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
I1007 17:00:50.754962 6532 mirrored_strategy.py:374] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 50000
I1007 17:00:50.754962 6532 config_util.py:552] Maybe overwriting train_steps: 50000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I1007 17:00:50.754962 6532 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W1007 17:00:50.817426 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['D:/my_tfrecord_data/train.tfrecord']
I1007 17:00:50.833063 6532 dataset_builder.py:162] Reading unweighted datasets: ['D:/my_tfrecord_data/train.tfrecord']
INFO:tensorflow:Reading record datasets for input file: ['D:/my_tfrecord_data/train.tfrecord']
I1007 17:00:50.833063 6532 dataset_builder.py:79] Reading record datasets for input file: ['D:/my_tfrecord_data/train.tfrecord']
INFO:tensorflow:Number of filenames to read: 1
I1007 17:00:50.848675 6532 dataset_builder.py:80] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W1007 17:00:50.848675 6532 dataset_builder.py:86] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\builders\dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
W1007 17:00:50.864345 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\builders\dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\builders\dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W1007 17:00:50.911177 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\builders\dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W1007 17:00:58.958072 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W1007 17:01:03.114336 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1007 17:01:06.911180 6532 deprecation.py:350] From C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\util\dispatch.py:1176: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
2023-10-07 17:01:10.883099: W tensorflow/core/framework/dataset.cc:769] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
C:\Users\square\anaconda3\envs\helmet\lib\site-packages\keras\backend.py:451: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
warnings.warn(
I1007 17:01:28.739308 9184 api.py:459] feature_map_spatial_dims: [(19, 19), (10, 10), (5, 5), (3, 3), (2, 2), (1, 1)]
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.004956 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.004956 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.004956 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.020551 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.020551 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I1007 17:01:31.020551 9184 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
I1007 17:03:58.074140 12100 api.py:459] feature_map_spatial_dims: [(19, 19), (10, 10), (5, 5), (3, 3), (2, 2), (1, 1)]
2023-10-07 17:07:23.316045: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1474560000 exceeds 10% of free system memory.
2023-10-07 17:07:26.577383: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1474560000 exceeds 10% of free system memory.
2023-10-07 17:07:31.681243: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1474560000 exceeds 10% of free system memory.
2023-10-07 17:07:33.887924: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 4423680000 exceeds 10% of free system memory.
2023-10-07 17:07:46.553382: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1658880000 exceeds 10% of free system memory.
2023-10-07 17:08:35.542556: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint: UNKNOWN: NewRandomAccessFile failed to Create/Open: C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint : Access is denied.
; Input/output error
Traceback (most recent call last):
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 92, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unable to open table file C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint: UNKNOWN: NewRandomAccessFile failed to Create/Open: C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint : Access is denied.
; Input/output error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\square\Tensorflow\models\research\object_detection\model_main_tf2.py", line 114, in <module>
tf.compat.v1.app.run()
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\platform\app.py", line 36, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\absl\app.py", line 308, in run
_run_main(main, args)
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\absl\app.py", line 254, in _run_main
sys.exit(main(argv))
File "C:\Users\square\Tensorflow\models\research\object_detection\model_main_tf2.py", line 105, in main
model_lib_v2.train_loop(
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\model_lib_v2.py", line 605, in train_loop
load_fine_tune_checkpoint(
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\object_detection\model_lib_v2.py", line 407, in load_fine_tune_checkpoint
ckpt.restore(
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\checkpoint\checkpoint.py", line 1977, in restore
status = self._saver.restore(save_path=save_path)
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\checkpoint\checkpoint.py", line 1448, in restore
reader = py_checkpoint_reader.NewCheckpointReader(save_path)
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 96, in NewCheckpointReader
error_translator(e)
File "C:\Users\square\anaconda3\envs\helmet\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 40, in error_translator
raise errors_impl.DataLossError(None, None, error_message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint: UNKNOWN: NewRandomAccessFile failed to Create/Open: C:\Users\square\Desktop\ssd_mobilenet_v2_320x320_coco17_tpu-8\checkpoint : Access is denied.
; Input/output error
I am trying to train a pre trained model "ssd_mobilenet_v2_320x320_coco17_tpu-8.config" using tf object detection API we have to give the file path of the fine_tuned checkpoints in this .config file i downloaded those from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md?plain=1 and i am using "SSD MobileNet v2 320x320" for some reason its constantly gving me access denied besides i have checked file permissions, running as administrator changed the paths but the problem remains constant. Please help with this