How to use beanstalkc in Python to queue URLs and perform jobs

2.2k Views Asked by At

I have a function named spider which takes seed as an argument. seed is the name of the URL I send to the spider function. Now my question is how do I use beanstalkc in Python to queue the URLs and perform the jobs.

1

There are 1 best solutions below

2
On

According to the tutorial you would need:

  1. beanstalkd server is running.
  2. Connect:

    import beanstalkc
    beanstalk = beanstalkc.Connection(host='localhost', port=14711)
    
  3. Add jobs using:

    beanstalk.put('seed url')
    
  4. Get job via:

    job = beanstalk.reserve()
    spider(job.body)
    
  5. Mark job as completed:

    job.delete()