Java - Python shared memory communication

Question

We have one program in Java and one in Python, and need to get them taking together in a ping-pong manner, each time exchanging an integer array of length 100,000, and taking ~ 0.1 - 1 second to do their work:

1. Java does some work and fires an int array of length 100,000 over to ...
2. Python, which does some work and fires a new array of length 100,000 back to ...
3. Java, which does some work ... etc

Note that

• Each program needs to wait for the other to do it's part.
• They will run on the same Linux machine.
• We will do Monte Carlo simulation, so speed is important.

I am more familiar with Java, and understand that a shared memory backed file approach is likely to be the fastest. This seems relevant for the Java side, but how would I get each program to wait/block for the other to complete its work and update the shared memory before the other starts reading? I've heard of something called 'semaphore', but can't figure it out.

These are my fallback ideas, but perhaps they are better?

• Unix Domain Sockets with jnr-unixsocket
• Sockets with Speedus

Answer 1

You could try combining java and python in the same process using jep. The latest release added support for sharing memory between python and java using numpy ndarrays and java direct buffers. This would let you share the data without any copying which should give the best performance possible.

Answer 2

Go for a fast intermediary data server to assist in communication between them. Redis would do the trick. You'll need two data structures there:

1. a list (your list of 100,000 items). We'll call that my_project:list for reference.
2. a lock. This can just be a Redis string set to "Python" or "Java."

Then have the following interaction:

1. Both Python and Java poll the Redis lock. If it's equal to "Python", it's Python's turn. If "Java," it's Java's turn.
2. Whichever program's turn it is goes into work mode and does whatever it needs to my_project:list, then it sets the lock to the other program's turn.
3. Repeat indefinitely.