R, Python and pyRserve - multi-threaded examples?

Question

I have a Python script set up where it instantiates Rserve, sets up a few R scripts and functions and then runs some data against the functions. However, I have been unable to create a multi-threaded instance of this same process. My core issue is that one thread always seems to dominate the processing and all of the other threads are ignored.

I've made the assumption that pyRserve can be multi-threaded - is that a correct assumption? Are there any examples out there that show this as a multi-threaded app?

Answer 1

I assume you were using a Windows machine. The news section of Rserve states for the most recent version (1.8-6) that

It is still strongly discouraged to use Windows given how limited the OS is and how it prevents any sensible use (Windows only supports single-client, single-thread, cooperative mode - so for toy uses only).

The Rserve webpage has more notes on Windows here:

Since the Windows operating system doesn't support fork method for spawning copies of a process, it is not possible to initialize R and use initialized copies for all subsequent connections in parallel. Therefore the Rserve for Windows supports no concurrent connections. This implies that all subsequent connections share the same namespace and sessions (as in >=0.4 version on unix) cannot be supported. It is still possible to start multiple Rserves to handle multiple connections (just make sure you use different port for each one).

However, I got it working on Windows with the Rserve version that comes with deployr-rserve (without having to use multiple processes running on different ports). This fork seems to be based on Rserve 0.6-3.1 though - so it's pretty outdated...

Answer 2

I kept on working with the code and it turns out that each thread needs its own port in order to work. I didn't find that documented anywhere, I was just trying out different idea. So:

1. I set up as many instances of Rserve as a I wanted threads. Each one of those instances was its own port
2. In my python code, when I instantiated the pyRserve object, I assigned it a unique port number.

Multi-threading is now working as desired and fast!