Dynaconf Object Access Fails After Loading Configuration from S3 Unless Iterated Over

Question

I'm encountering a strange issue while loading TOML configuration files from an AWS S3 bucket into a Dynaconf object in Python.

Here's a simplified version of the code I'm using:

import os
import boto3
from dynaconf import Dynaconf

def load_settings(template_name: str) -> Dynaconf:
    s3 = boto3.client("s3")
    key = f"{template_name}.toml"
    obj = s3.get_object(Bucket="my_bucket", Key=key)
    toml_str = obj["Body"].read().decode("utf-8")

    temp_file = f"{template_name}.toml"

    # Write the TOML string to the temporary file

    with os.fdopen(fd, "w") as file:
        file.write(toml_str)

    settings = Dynaconf(
        envvar_prefix="DYNACONF",
        environments=True,
        settings_files=[temp_file]
    )
    
    # Iterating over the items
    for k, v in settings.items():
        print(k, v)

    # Now I can access the values
    print(settings.my_value)

    os.remove(temp_file)
    return settings

The problem arises when I try to directly access a value from the settings object (for example, settings.my_value) after loading the configuration from the S3 bucket. This direct access fails unless I previously iterate over the items in settings.

Expected behavior: I should be able to directly access a value from the settings object without first iterating over all the items.

Actual behavior: Direct access fails with an error message stating that the requested key doesn't exist, unless I first iterate over the items in settings.

This is particularly puzzling because if I comment out the iteration over the items in settings, the print statement fails, stating that 'my_value' doesn't exist. But, if I leave the iteration in place, the print statement succeeds.

Any ideas why this might be happening? Is there something about how Dynaconf loads or accesses data that I'm missing here? Any help would be greatly appreciated!

Update: even better, give me a guideline on what would be the proper way of loading a remote settings file.

Answer 1

Dynaconf object access issue might be due to lazy loading. You can use Dynaconf's remote function to load settings from S3 and merge them with your existing settings. This way, you can directly access the values without iterating over the items first. I've provided below code you can check and let me know.

from dynaconf import Dynaconf

settings = Dynaconf(
    envvar_prefix="DYNACONF",
    environments=True,
)

settings.load_remote("s3://my-bucket/my-settings.toml")

# Now you can access the settings.my_value below
print(settings.my_value)

Answer 2

To begin with, let me point out a few minor issues. I do not think your usage of get_bucket is correct as it states otherwise in boto3 docs. Second, os.fdopen is redundent. Simply use open with the same parameters. Third, there is no need to wrap the file's name in a list. As in Dynaconf docs, a single string with a single filename is okay.

Now, as for your problem, you put the configuration under a scope, didn't you? dynaconf lazyloads those and put the SCOPES as properties and not the keys. Putting the configuration outside of a scope solves everything E.g.

my_value = 1

instead of

[default]
my_value=1

If you MUST use a scope, write settings.SCOPE_NAME.my_value. I don't really know why iterating helps but here is the solution. If it helps, please click upvote&accept&bounty, thank you!