Multi-threading when using swi prolog as embeded engine

Help!
#1

I’m using: SWI-Prolog version 8.03.

We need to run our application in parallel. We used “Multithreaded mixed C and Prolog applications” – https://www.swi-prolog.org/pldoc/man?section=foreignthread. I want the code to share some big relations across the threads. Meanwhile, we also need to create smaller and unique relations in each thread. Each thread needs to run the queries against all the relations, small and big.

I tried to allocate threads through C (Python actually, but under the hood, it’s the same). I then assign each thread a separate prolog engine using PL_thread_attach_engine(NULL) function. However, when I run the query, I always get the “nested query” error - “NestedQueryError: The last query was not closed”.

Is it possible to do this?

TIA.
Jackson

#2

SWI-Prolog can definitely do this. The NestedQueryError should come from the Python interface as this term doesn’t exist in the SWI-Prolog source. This smells like the Python interface is using global variables. Note that where SWI-Prolog is fully multi-threaded, Python is a bit half way, using a central lock. At least that was the situation when I asked one of the early developers of Python about locking requirements when calling Python from C.

#3

Thanks Jan! You are right. I found the central lock in the python interface and disabled it for now. Everything works perfectly now.

Please let me know if I do need to add some mutex to the application to coordinate the prolog engines. Otherwise, all is good. Thanks!

#4

Prolog engines should be fine. I’d be more worried about disabling the Python global lock. You might consider running Prolog as a web server using either plain HTTP, plain sockets or Pengines. All have their pros and cons. Separating Prolog has mostly cons except if you cannot tolerate the network latency.

#5

Is this the Python Global Interpreter Lock (GIL) or something specific for your Python interface?

#6

No. It’s not the GIL, but a specific thing in this specific python wrapper.

#7

Thanks Jan for the advice. To be more informative to the people in similar situation, I am using pyswip, a python wrapper(bridge) of swi-prolog. Pyswip is based on ctypes. I made some changes to pyswip to add the support to attach a prolog engine to each thread.

For the “nested query” problem I was talking about, I disabled the class variable of Prolog. It’s not the GIL. Disabling GIL even temporarily is too risky.

1 Like
#8

There may be a good reason why pyswip has a lock to enforce single-threaded Prolog querying. (I ran into a problem with an earlier pyswip version once when I tried to use it with Python multiprocessing.)

To be safe, you might want to pass goals to Prolog asynchronously and have Prolog create threads and send goals to them through message queues, for example. When a result gets sent back, your main Prolog thread can call back into Python with the result. In that way pyswip is only “aware” of a single thread.

#9

Thanks for the info. I admit that I haven’t tested my solution thoroughly. There might be some pitfalls that I am not aware of. I will have to wait for that to happen before I can address them. I am pretty sure multi-processing is totally different from my multi-threading approach. Processes don’t share any data (unless you specifically let them), while threads do. So the problem that you ran into maybe different. However, without more details I can not say for sure.

I don’t know how to use swi prolog’s C interface to let prolog create threads, except that writing special directives in prolog program and then use “consult” or sth like that. I will dig deeper if I can find more info about that. So far I have been using python to create threads, which I believe is the same as using C to create OS threads.

#10

Hi,
@freedafeng can you please share a few lines of code? I am also trying to modiy the prolog.py file to not to get nested query error. Which lines did you commented/changed? I have been trying to solve this problem for a few days. This is very urgent to me. I would appreciate if you can help. Many thanks in advance…

Nested Query
#11

Hello Jan,
Can you please clarify how to fix this problem?
Many thanks

#12

I’m afraid not. I have nothing to do with the Python interface and merely explained the error is not from Prolog (and thus from the interface). Hope one of the other can clarify.