Managing data storage in a linux application written in Python

Question

In the past, I've written some simple Python scripts and some web apps in Django. Now I've set my sights on a full fledged application that would run in Linux. There's thousands of tutorials on getting the GUI up and running, but I can't wrap my head around the storing the state of the application.

For simplicity's sake, say I want to keep track of a counter in my application. I have a button and every time the user clicks the button, the counter will be increased. How do I make sure that in case of a crash or the program exiting, the user doesn't lose all the "progress" on the counter? Should I be opening a file, writing to it, and closing it every time the user increments the counter? That seems like a lot of overhead. Should I be using databases? Is it more akin to a client/server relationship like a webpage would use?

Answer 1

In the example you gave; a user pressing a button that increments a counter some applications would:

1. Write to a file every time the counter is changed
2. Write to a file every 10 increments, or whatever number you want, anything so that it isn't constantly writing to a file
3. Do the first 2 just with a database.

If you are making an app that runs locally I would usually just write to a file. I would make checkpoints and when those checkpoints are hit you will save the app state to the files.

If your app will be running on multiple devices with different users you should probably use a database. Also using a database will act as a backup in case data on your machine got corrupted or something like that.

If you do decide to use a database I would recommend using mongodb, you can use it free and it is relatively simple to get working in python. A basic client would only be about 35 lines. Example

If you decide to store it locally there's a few things you could do. If you have a simple application you could just store the data in txt files i.e. counter.txt then you could write to this every few increments or decrements.

If your application has a more complicated storage structure you could store the entire app state as a object in python and to save the state you could simply use

with open("app_state.json", "w") as f:
    json.dump(app_state, f)

you would do this every time the application hits some sort of checkpoint. This could just be a timer that does it every 5 minutes

As you mentioned you could also use a SQLite database, which is a database that runs locally. This is a good way to manage data if you don't want to read and write to a bunch of files. This will also give you some security over the data.