error in loading pickle

Question

Not able to load a pickle file. I am using python 3.5

import pickle
data=pickle.load(open("D:\\ud120-projects\\final_project\\final_project_dataset.pkl", "r"))

TypeError: a bytes-like object is required, not 'str'

. .

Also tried:

import pickle
data=pickle.load(open("D:\\ud120-projects\\final_project\\final_project_dataset.pkl", "rb"))

UnpicklingError: the STRING opcode argument must be quoted

. .

Same errors even when using with statements

import pickle
with open("D:\\ud120-projects\\final_project\\final_project_dataset.pkl", "rb") as f:
    enron_data = pickle.load(f)

Answer 1

I'm using windows 10 and vscode, you should go to the final_project_dataset.pkl file and then change option CRLF to LF and then save the file then UnpicklingError: the STRING opcode argument must be quoted error will be disappeared.

change CRLF to LF

then save the final_project_dataset.pkl file.

Answer 2

You definitely need the "rb" to read the file, which solves the first problem.

The second issue (STRING opcode argument) is because the file doesn't have Unix line endings. You need to run the pkl file through a script to convert them. If you see this thread, there is a script called "dos2unix" that will solve that for you:

How to convert DOS/Windows newline (CRLF) to Unix newline (\n) in a Bash script?

Answer 3

The only Fix for me was(Answered by Monkshow92 in Github) :

" The pickle file has to be using Unix new lines otherwise at least Python 3.4's C pickle parser fails with exception: pickle.UnpicklingError: the STRING opcode argument must be quoted I think that some git versions may be changing the Unix new lines ('\n') to DOS lines ('\r\n').

You may use this code to change "word_data.pkl" to "word_data_unix.pkl" and then use the new .pkl file on the script "nb_author_id.py": dos2unix.txt

#!/usr/bin/env python
"""
convert dos linefeeds (crlf) to unix (lf)
usage: dos2unix.py 
"""
original = "word_data.pkl"
destination = "word_data_unix.pkl"

content = ''
outsize = 0
with open(original, 'rb') as infile:
    content = infile.read()
with open(destination, 'wb') as output:
    for line in content.splitlines():
        outsize += len(line) + 1
        output.write(line + str.encode('\n'))

print("Done. Saved %s bytes." % (len(content)-outsize))

dos2unix.py addapted from: http://stackoverflow.com/a/19702943

Small Tweak that I have found is , Changing "r" mode to "rb" byte object mode. And finally converting all the .pkl files using above python script to convert from Dos to Unix !

Answer Link : https://github.com/udacity/ud120-projects/issues/46 Full Credit : Monkshow92

Answer 4

If an entire script puts you off, it really just deserves one line:

import pickle
with open("D:\\path\\to\\file.data", "rb") as f:
    lines = [line.rstrip("\r\n") for line in f.readlines()]
    data = pickle.loads("\n".join(lines))