Python3 csv DictWriter: writing through bz2 fails (TypeError: 'str'/buffer interface)

Question

I am trying to write an on-the-fly bz2-compressed csv file in python3.

Without compression this works fine:

from csv import DictWriter

FIELDNAMES = ('a', 'b')
OUT_CSV = 'out.csv'

DATA = ({'a': 1, 'b': 2}, {'a':3, 'b':4})

# works fine
with open(OUT_CSV, 'w') as file_pointer:
    csv_writer = DictWriter(file_pointer, fieldnames=FIELDNAMES)
    csv_writer.writeheader()
    for dataset in DATA:
        csv_writer.writerow(dataset)

Adding compression it fails:

from csv import DictWriter
import bz2

FIELDNAMES = ('a', 'b')
OUT_BZ2 = 'out.csv.bz2'

DATA = ({'a': 1, 'b': 2}, {'a':3, 'b':4})

# this fails
with bz2.open(OUT_BZ2, mode='w', compresslevel=9) as file_pointer:
    csv_writer = DictWriter(file_pointer, fieldnames=FIELDNAMES)
    csv_writer.writeheader() # fails here; raises "TypeError: 'str' does not support the buffer interface"
    for dataset in DATA:
        csv_writer.writerow(dataset)

with the Exception: TypeError: 'str' does not support the buffer interface.

Is there a buffer-compatible csv module out there? Or do I have to write the csv rows by hand? or is there an elegant solution?

Answer 1

Note that the bz2.open defaults to binary mode. This is in contrast to the regular Python built-in open which defaults to text mode. To solve your error, you simply need to open the file in text mode, not the default binary mode. Change the with bz2.open(... mode='w'...) to with bz2.open(... mode='wt'...). I tested on Python 3.4.