My question is related to uploading multiple fields from the scrapy.Item
value that we get to S3 buckets. I see that it is possible to use feed exports to write all items to a single file with various formats by specifying the bucket url. My requirement is as below:
If my item has multiple fields as below, I would like to upload each of these fields (which can be html or md files) as separate objects to S3 buckets. I found something similar where I could write a custom exporter using the boto3
package for the s3 connection and do a per item upload as seen here: https://stackoverflow.com/a/41973194
from scrapy.item import Item, Field
class CustomItem(Item):
field_one = Field()
field_two = Field()
My question was whether there is a more native way to do this in Scrapy such as using feed exports or if there is any better way than writing custom exporters.