How do I use secrets in Iguazio?

Question

How do I define secrets in Iguazio? Are they cluster-wide or just at the project level? Once the secret is defined, how do I use it in my jobs? What about distributed jobs that have many pods/workers like Spark and Dask, can I use secrets with those types of frameworks?

Thanks.

Answer 1

You can define a secret in Iguazio. You can either define a secret at the project level or at the cluster level. For define a secret at the project level, here is an example code snippet:

# Create project secrets for the myproj project
project = mlrun.get_or_create_project("myproj", "./")
secrets = {'AWS_KEY': '111222333'}
project.set_secrets(secrets=secrets, provider="kubernetes")

# Create and run the MLRun job
function = mlrun.code_to_function(
    name="secret_func",
    filename="my_code.py",
    handler="test_function",
    kind="job",
    image="mlrun/mlrun"
)
function.run()

MLRun provides facilities to map k8s secrets that were created externally to jobs that are executed.

function = mlrun.code_to_function(
    name="secret_func",
    handler="test_function",
    ...
)

function.set_env_from_secret(
    "SECRET_ENV_VAR_1", secret="my-secret", secret_key="secret1"
)

Then you will use the secret in a function like this:

# Function handler
def test_function(context):
    # Getting the value in the secret2 key.

    my_secret_value = os.environ.get("SECRET_ENV_VAR_2")

    ...