TypeError: can't pickle CompiledFFI objects

Question

I am trying to get output from Telnet and SSH hosts for some commands and store them in a shelf. Since there are many commands, I am using multiprocessing. I have the following important methods:

1. connectToHost: Making a connection (SSH/ Telnet) using a method.
2. ExecuteCommand: executing the command and getting the output.
3. main: where I am iterating over all the commands and getting the output in a dictionary using multiprocessing. This method calls another method which merges the outputs from other processes.

For Telnet, everything works fine but when it is an SSH connection, I the code is failing at the process.start() and I am getting following error.

TypeError: can't pickle CompiledFFI objects

Why is it not working for SSH even when I am calling the same methods? how to fix this?

Answer 1

When you start a new process python has to pass all variables used for that process. In this case it is a connection to a certain host. Variables has to be serialized (pickled) in order to do that.

It looks like paramiko uses FFI which objects can not be pickled.

Problem can be solved by changing multiprocessing into multithreading (pickle won't be needed) or by creating ssh connection in every process from the beginning (which may not be so efficient).