Erlang->C++->Python->C/C++ Import Error: undefined symbol: PyObject_SelfIter

Questions / Help
,
1

I’m trying to run a Python function from Erlang using a C++ driver. I am able to run the Python function outside of Erlang by calling the C++ driver function and everything works perfectly, but when I try to call it from Erlang, I get the following import error:

ImportError: /######/lib/python3.8/site-packages/grpc/_cython/cygrpc.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter

This error happens when the C++ driver runs the PyImport_Import() function on the Python file. It looks like the shared library mentioned in the error is trying to access the “Python.h” file and can’t find it. I tried adding its location to the PATH environment variable (using Py_SetPath()) but that did not solve the issue.

My C++ driver uses the “Python.h” file, but I have to include the path at compile time or else I get a similar import error.

Is there a specific environment variable I need to set in my rebar3.config file or elsewhere to allow the mentioned shared library to find the “Python.h” file? Or is there something else I’m doing (or not doing) that is causing this error?

2 Likes
2

It sounds like your LD_LIBRARY_PATH simply doesn’t have /######/lib/python3.8/site-packages/grpc/_cython/ on it. Make sure this is set before launching from erlang.

1 Like
3

@starbelly, thanks for responding. I did try setting that variable in my C++ driver (using setenv) before running Py_Initialize(), but that didn’t fix the issue. Do I need to set it in a different place?

1 Like
4

You need to set in your shell before you boot erlang, that’s how I’ve always handled these issues.

1 Like
5

Ok, I tried running export LD_LIBRARY_PATH=/######/lib/python3.8/site-packages/grpc/_cython/ before running rebar3 shell, but I’m still getting the error. I also tried doing the same thing but setting LD_LIBRARY_PATH to the location of Python.h but that didn’t work either.

1 Like
6

QQ: I’m assuming you didn’t literally use /######/... instead you used ###### in this thread to obfuscate a path on your system. Is that correct?

1 Like
7

That is correct. Speaking of which, is there a better way to indicate this?

1 Like
8

Not really :slight_smile: Maybe say so next time to be clear. It sounds like you’re still missing some libs on your LD_LIBRARY_PATH. Now that I look back at the error, you’re missing part of python itself on your LD_LIBRARY_PATH. This could also be due to linking problems. I’d do some googling on the error :slight_smile:

1 Like
9

I have spent about a week googling the error and trying various things with no success. What makes me think it is Erlang related is the fact that the code runs fine outside of Erlang (calling the driver functions directly from C++). It’s when I try to run it from inside Erlang that I get the error.

1 Like
10

The error indicates that libpython is not loaded. How do you usually run this code? What does ldd your_driver.so show?

1 Like
11

Here’s the output of ldd:

	linux-vdso.so.1 (0x00007ffe9b5cc000)
	libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f8196ccd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8196adb000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f8196aad000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8196a91000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8196a6e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8196a68000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f8196a61000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8196912000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8197248000)

What’s weird is the fact that my C++ driver uses libpython to setup and start the Python interpreter, so it has to be loaded. But it seems as though inside the Python interpreter, it can’t access it.

In normal operation (from Erlang), I load the driver, start the driver, and then do a port_control call to execute Python call, although I haven’t gotten past starting the driver, since that’s where I run into the issue.

The C++ driver just returns 0 for the init call. In the start call, it does a Py_SetPath, Py_Initialize, and then attempts the PyImport_Import at which point it runs into the import error. I have tried adding the path to libpython in the Py_SetPath command as well, with no luck.

And just to reiterate, when I run the C++ driver outside of Erlang, all is well. But when I try to run it from Erlang, I get this issue.

1 Like