How can I periodically run a Python script to import data into a Django app?

Question

I have a script which scans an email inbox for specific emails. That part's working well and I'm able to acquire the data I'm interested in. I'd now like to take that data and add it to a Django app which will be used to display the information.

I can run the script on a CRON job to periodically grab new information, but how do I then get that data into the Django app?

The Django server is running on a Linux box under Apache / FastCGI if that makes a difference.

[Edit] - in response to Srikar's question When you are saying " get that data into the Django app" what exactly do you mean?...

The Django app will be responsible for storing the data in a convenient form so that it can then be displayed via a series of views. So the app will include a model with suitable members to store the incoming data. I'm just unsure how you hook into Django to make new instances of those model objects and tell Django to store them.

Answer 1

When you are saying " get that data into the DJango app" what exactly do you mean?

I am guessing that you are using some sort of database (like mysql). Insert whatever data you have collected from your cronjob into the respective tables that your Django app is accessing. Also insert this cron data into the same tables that your users are accessing. So that way your changes are immediately reflected to the users using the app as they will be accessing the data from the same table.

Answer 2

Best way?

Make a view on the django side to handle receiving the data, and have your script do a HTTP POST on a URL registered to that view.

You could also import the model and such from inside your script, but I don't think that's a very good idea.

Answer 3

You can write custom admin command to load data according to your need and run that command through cron job. You can refer Writing custom commands

You can also try existing loaddata command, but it tries to load data from fixture added in your app directory.

Answer 4

I think Celery is what you are looking for.

Answer 5

Have your script send an HTTP Post request like so. This is the library Requests

>>> files = {'report.xls': open('report.xls', 'rb')}

>>> r = requests.post(url, files=files)
>>> r.text

then on the receiving side you can use web.py to process the info like this

x = web.input()

then do whatever you want with x

On the receiving side of the POST request import web and write a function that handles the post

for example

    def POST(self):

        x = web.input()

Answer 6

If you dont want to use HTTP to send messages back and forth you could just have the script write the email info to a .txt file and then have your django app open the file and read it.

EDIT:

You could set your CRON job to read the e-mails at say 8AM then write it to a text file info.txt. The in your code write something like

import time
    if '9' == time.strftime("%H"):
        file = open(info.txt)
        info = file.read()

that will check the file at 9AM untill 10AM. if you want it to only check one time just add the minutes too the if statement as well.

Answer 7

I have done the same thing.

Firstly, my script was already parsing the emails and storing them in a db, so I set the db up in settings.py and used python manage.py inspectdb to create a model based on that db.

Then it's just a matter of building a view to display the information from your db.

If your script doesn't already use a db it would be simple to create a model with what information you want stored, and then force your script to write to the tables described by the model.

Answer 8

Forget about this being a Django app for a second. It is just a load of Python code.

What this means is, your Python script is absolutely free to import the database models you have in your Django app and use them as you would in a standard module in your project.

The only difference here, is that you may need to take care to import everything Django needs to work with those modules, whereas when a request enters through the normal web interface it would take care of that for you.

Just import Django and the required models.py/any other modules you need for it work from your app. It is your code, not a black box. You can import it from where ever the hell you want.

EDIT: The link from Rohan's answer to the Django docs for custom management commands is definitely the least painful way to do what I said above.