ClearML how to change clearml.conf file in AWS Sagemaker

Question

I am working in AWS Sagemaker Jupyter notebook. I have installed clearml package in AWS Sagemaker in Jupyter. ClearML server was installed on AWS EC2. I need to store artifacts and models in AWS S3 bucket, so I want to specify credentials to S3 in clearml.conf file. How can I change clearml.conf file in AWS Sagemaker instance? looks like permission denied to all folders on it. Or maybe somebody can suggest a better approach.

Answer 1

Disclaimer I'm part of the ClearML (formerly Trains) team.

To set credentials (and clearml-server hosts) you can use Task.set_credentials. To specify the S3 bucket as output for all artifacts (and debug images for that matter) you can just set it as the files_server.

For example:

from clearml import Task

Task.set_credentials(api_host='http://clearml-server:8008', web_host='http://clearml-server:8080', files_host='s3://my_bucket/folder/',
key='add_clearml_key_here', secret='add_clearml_key_secret_here')

To pass your S3 credentials, just add a cell at the top of your jupyter notebook, and set the standard AWS S3 environment variables:

import os
os.environ['AWS_ACCESS_KEY_ID'] = 's3_bucket_key_here'
os.environ['AWS_SECRET_ACCESS_KEY'] = 's3_bucket_secret_here'
# optional
os.environ['AWS_DEFAULT_REGION'] = 's3_bucket_region'