can't call julia code from Python with PyCall, whose code calls python back with Pycall

Question

I have a backend algorithm which makes julia calls, I do this via the PyCall package. however, one of the packages in julia recently updated, and it itself makes use of PyCall. no mater what I try to do, I always get an error. If i set pycall in julia via Pkg.build("PyCall"), I can call the package, however python cannot call julia. if I set pycall in python3 via julia.install(), then the package can't run properly in julia.

to reproduce this here is a quick example:

test.jl

using Pkg
Pkg.add("PyCall")
ENV["PYTHON"]=""
Pkg.add(PackageSpec(url="https://bitbucket.org/SSamanipour/safd.jl/src/master/"))
Pkg.build("SAFD")
Pkg.test("SAFD")

test.py

from julia import Main as jl
jl.include("test.jl")

if I run python3 test.py

I get into an error as that specific python is not configured py the shared pycall.

what I would like is that the pycall in python not be the same for python. it seems that if you configure pycall in one it will force the other.

I tried using juliacall, however if I use pycall with it, it seems to also break.

if I try to have ENV["PYTHON"]="python3", I can't run the SAFD package.

Any help would be greatly appreciated, as I've been a few days on this and tried multiple solutions and everything seems to give me a different error.

some info: doing this on an aws amazon2 linux OS and an aws r5.large instance.

I'm using python 3.10.0

versioninfo() Julia Version 1.6.3 Commit ae8452a9e0 (2021-09-23 17:34 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)

EDIT:

I ended up figuring out the issue, The package I was trying to use updated its dependencies, when it builds, it automatically updates the conda python with all the packages it needs. However, if it is not built by conda, it will assume the dependencies are there. What I'm very surprised by is that the error message never indicated anything about a specific package. I only figured it out a bit randomly.

One thing is clear is that you can't have two different pythons, pycall only configures one python for back and forths between python and julia.

Answer 1

You seem to have 2 Python's on your system:

• first one installed after ENV["PYTHON"]="";Pkg.build("PyCall") --> this is the default way to install PyCall in Julia to get everything working (this is actually a Miniconda Python)
• system Python that you start with python3

Basically, mixing those 2 Pythons and having everything to work is like finding a needle in a haystack being blindfolded. Rather than that you need to use one Python for everything and to get things running smoothly that should be the Miniconda Python that got installed by Julia when doing ENV["PYTHON"]="";Pkg.build("PyCall").

In order to find Python you want to use do in your Julia console:

julia> using PyCall

julia> PyCall.python
"/home/ubuntu/.julia/conda3/python.exe"

Now you found the Python code that you need! So what you run in bash is:

/home/ubuntu/.julia/conda3/python.exe test.py