Removing a non-nullable Django field with SeparateDatabaseAndState

Question

Let's say I have the following model:

class Product(models.Model):
    name = models.CharField(max_length=128)
    is_retired = models.BooleanField(default=False)

I want to remove the is_retired field. I'm using blue-green deployments to release changes to production so I'm using SeparateDatabaseAndState in my migration.

My initial migration to remove the field from the application state is simple:

class Migration(migrations.Migration):
    dependencies = ...
    operations = [
        migrations.SeparateDatabaseAndState(
            state_operations=[
                migrations.RemoveField(
                    model_name="product",
                    name="is_retired",
                ),
            ],
            database_operations=[],
        ),
    ]

I can successfully run the migration. However, I'm getting the following error when I attempt to create a new product with Product.objects.create(name="Wrench"):

self = <django.db.backends.sqlite3.base.SQLiteCursorWrapper object at 0x106bf4050>
query = 'INSERT INTO "core_product" ("name") VALUES (?) RETURNING "core_product"."id"'
params = ('Wrench',)

    def execute(self, query, params=None):
        if params is None:
            return super().execute(query)
        # Extract names if params is a mapping, i.e. "pyformat" style is used.
        param_names = list(params) if isinstance(params, Mapping) else None
        query = self.convert_query(query, param_names=param_names)
>       return super().execute(query, params)
E       django.db.utils.IntegrityError: NOT NULL constraint failed: core_product.is_retired

Looks like the INSERT query is failing because the is_retired field is defaulting to null, instead of its correct default of False.

I fixed it by making Product.is_retired nullable before removing it from the state:

class Migration(migrations.Migration):
    dependencies = ...
    operations = [
        migrations.AlterField(
            model_name="product",
            name="is_retired",
            field=models.BooleanField(default=False, null=True),
        ),
        migrations.SeparateDatabaseAndState(
            state_operations=[
                migrations.RemoveField(
                    model_name="product",
                    name="is_retired",
                ),
            ],
            database_operations=[],
        ),
    ]

Now I can successfully create a product with Product.objects.create(name="Wrench").

A few questions:

• Is this the best way to fix this error? Any other ideas? I looked into changing the is_retired default back to False but couldn't figure it out.
• Is changing a field from not nullable to nullable safe during a blue-green deployment? I'm pretty sure it is but just want to make sure.

Answer 1

Your approach to making the is_retired field nullable before removing it from the state is a valid solution to the problem you encountered. However, let's address your questions directly:

Is this the best way to fix this error? Any other ideas?

Your approach is a sensible one. By making the field nullable before removing it from the state, you avoid the NOT NULL constraint failure when inserting new records. Another approach could involve manually setting the default value of the field in the database before running the migration to remove it from the state, but this might involve more manual database operations and could be error-prone.

Is changing a field from not nullable to nullable safe during a blue-green deployment?

Changing a field from not nullable to nullable should generally be safe, especially if your database can handle null values gracefully. In your case, since you're using a blue-green deployment strategy, the risk is mitigated further as you can test the changes in a separate environment before switching traffic to it. However, it's always a good practice to thoroughly test such changes to ensure they don't have unintended consequences on your application's behavior.

Answer 2

Your approach is essentially correct and would work for most scenarios. As far as I know, it is not dangerous either. However, depending on the importance of the model field in your application and your feature release strategy in the blue/green deployment, you may want to increase control over the is_retired model field.

The idea of a feature flag in the case of a blue/green deployment is interesting because it allows you to trigger the use or not of a specific functionality at a given time. In the case of is_retired, you may want to have it active for some users while gradually giving it up across your entire application(s).

Here is a basic implementation:

in your settings.py

FEATURE_FLAGS = {
    'IS_RETIRED_ENABLED': os.getenv('IS_RETIRED_ENABLED', 'False') == 'True',
}

Working with an environment variable is important because it is what will allow you to trigger different behavior for the feature in different places.

then in your models.py

class Product(models.Model):
    name = models.CharField(max_length=128)
    is_retired = models.BooleanField(default=False, null=True)

    @staticmethod
    def create_product(name, is_retired=None):
        if settings.FEATURE_FLAGS['IS_RETIRED_ENABLED']:
            return Product.objects.create(name=name, is_retired=is_retired)
        else:
            return Product.objects.create(name=name)

technically you could also place this function your views.py or anywhere but in most case, keeping the logic close to the model is the way to go.

Finally, across your various environments, for which you wish to use or not the is_retired field, you can do :

export IS_RETIRED_ENABLED=True

or 

export IS_RETIRED_ENABLED=False

In conclusion, this is more involved than setting the is_retired to null before the migration process and might not be appropriate for some cases, especially for not so important fields but it is good to know the option is available when you need more flexibility.