Django -- use some production data in test database

Question

My project is using Django's test database framework (w/ and w/o Selenium) and it's been working well for several years, with a few read-only tests on the production database (mostly integrity tests that our DB can't enforce) but with the vast majority being R/W on the test database.

We have one model/table on the production database that provides important metadata for the site which is getting too big to code into the fixtures, and which we would like to see current values for in our tests. I would love our setUp() code to be able to do something like:

def setUp(self):
    with self.activate_production_database():
         metadata_info = MetadataTable.objects.values_list(
             'title', flat=True)

    # back to test_database
    MetadataTable.objects.bulk_create([MetadataTable(title=t)
                                       for t in metadata_info])

I'm wondering what, if anything, exists that is like the with self.activate_production_database() line?

Answer 1

Whatever you do, do NOT allow cross read on the production database from other (non prod) systems. It might be fine for this one use case, but the usage will bleed out into other things as people realise it exists, and you will end up with problems.

Some possible solutions:

• Have a replicated read-only database containing only the table(s) you want to read, and use this.
• It feels this data is part of the system, rather than data, so take it out of the table and include it in the repo as eg. json, yaml etc. Then load (automatically) it to the table each time you do a deploy - this means your other environments will stay up to date as long as you have the latest repo.
• Put this common data into a SQLite database instead and have django read 2 databases. Include the SQLite db file in the repo.

I think what you're doing rn is as I mentioned above - you've put system variables that are the same in all environments into the database, when they need to be part of the system. So the solution is to find a sensible way to include them in the system and have them as part of the repo, even if this also means the deployment process recreates the existing table each time.