We use the libzdb library in one of our projects together with mysql 5.5
This program is multithreaded and the different threads (forks) use the same connection pool. On a sigterm to the main process the child processes are killed and the main process will clear all allocated resource. When this main process calls "ConnectionPool_free" this function never returns.
I can produce more debug info when needed.
Regards,
On Jul 19, 2011, at 3:27 PM, Michel Verbraak wrote:
We use the libzdb library in one of our projects together with mysql 5.5
This program is multithreaded and the different threads (forks) use the same connection pool. On a sigterm to the main process the child processes are killed and the main process will clear all allocated resource. When this main process calls "ConnectionPool_free" this function never returns.
I can produce more debug info when needed.
If you ask if we have seen the same problem, then the answer is no. In our own applications which uses libzdb we also call ConnectionPool_free after receiving SIGTERM. Calling ConnectionPool_free is part of a shutdown chain and we should have noticed if the call did not return.
When you say "threads (forks)", I wonder if you use libzdb from actual sub-processes created via fork instead of threads? If this is the case, then your problem is right there - libzdb use mutexes to serialize access to shared resources which only works with threads.
On 19-07-11 17:55, Jan-Henrik Haukeland wrote:
On Jul 19, 2011, at 3:27 PM, Michel Verbraak wrote:
We use the libzdb library in one of our projects together with mysql 5.5
This program is multithreaded and the different threads (forks) use the same connection pool. On a sigterm to the main process the child processes are killed and the main process will clear all allocated resource. When this main process calls "ConnectionPool_free" this function never returns.
I can produce more debug info when needed.
If you ask if we have seen the same problem, then the answer is no. In our own applications which uses libzdb we also call ConnectionPool_free after receiving SIGTERM. Calling ConnectionPool_free is part of a shutdown chain and we should have noticed if the call did not return.
When you say "threads (forks)", I wonder if you use libzdb from actual sub-processes created via fork instead of threads? If this is the case, then your problem is right there - libzdb use mutexes to serialize access to shared resources which only works with threads.
-- To unsubscribe: http://www.tildeslash.com/mailman/listinfo/libzdb-general
We use fork and not threads. So the problem is clear. Thank you.
On Jul 20, 2011, at 8:32 AM, Michel Verbraak wrote:
We use fork and not threads. So the problem is clear.
I should qualify my previous statement a bit; Using libzdb from several sub-processes at the same time should not a problem per se if each sub-process create its own connection pool. The problem arise if the main thread/process initialize libzdb and then call fork. I suspect this is what's happening. Pthread structures does work well with fork. Especially mutexes can be left in an inconsistent state. For instance, if a mutex was locked in the parent process when it forked, the mutex will continue to be locked in the child process and there is no way for the child to unlock the mutex since the owner is a non-existing thread. ConnectionPool_free will lock the pool before it drains it and if the mutex is in an inconsistent state, calling ConnectionPool_free will typically hang on mutex lock and therefor never return.
We should be able to set libzdb in a proper state for a child process by using pthread_atfork(), but until we decide to implement this; if each sub-process create its own connection pool using ConnectionPool_new you should be fine.
Jan-Henrik
On 20-07-11 14:27, Jan-Henrik Haukeland wrote:
On Jul 20, 2011, at 8:32 AM, Michel Verbraak wrote:
We use fork and not threads. So the problem is clear.
I should qualify my previous statement a bit; Using libzdb from several sub-processes at the same time should not a problem per se if each sub-process create its own connection pool. The problem arise if the main thread/process initialize libzdb and then call fork. I suspect this is what's happening. Pthread structures does work well with fork. Especially mutexes can be left in an inconsistent state. For instance, if a mutex was locked in the parent process when it forked, the mutex will continue to be locked in the child process and there is no way for the child to unlock the mutex since the owner is a non-existing thread. ConnectionPool_free will lock the pool before it drains it and if the mutex is in an inconsistent state, calling ConnectionPool_free will typically hang on mutex lock and therefor never return.
We should be able to set libzdb in a proper state for a child process by using pthread_atfork(), but until we decide to implement this; if each sub-process create its own connection pool using ConnectionPool_new you should be fine.
Jan-Henrik
To unsubscribe: http://www.tildeslash.com/mailman/listinfo/libzdb-general
Jan-Hendrik,
We create one pool in the main process before any fork is done. This pool is freed when the main process terminates. The main process sends a sigterm to all child processes when it receives a sigterm. The child process does an exit when it receives the sigterm.
We cannot let each child process create it's own pool because the main process is a listening server and on a new connection a fork is done. Currently we do not use ConnectionPool_free because the whole process is ended on a sigterm and memory is released. It is dirty but works until we get our software right.
Regards,
Michel
On Jul 20, 2011, at 3:57 PM, Michel Verbraak wrote:
We cannot let each child process create it's own pool because the main process is a listening server and on a new connection a fork is done.
A child process duplicates the image of the parent process at fork time so the child process will have its own connection pool whether you want it to or not. Or rather, it will have a copy of the parents connection pool at fork time. And since ConnectionPool_free hangs, the pool is probably in an inconsistently state. So, given that your child processes will have its own pool anyway, it is better to make sure that the pool they have works correctly. This is done by moving the Connection Pool creation and initialization code to the code the child execute first after a fork. A simpler option, unless you must for some reason use sub-processes, is to replace sub-processes with threads. Libzdb is intended to be used and works great in a multi-threaded application.
If you don't want to change any code now, at least make sure that the connection pool is _not_ started with a reaper thread, because the reaper locks the pool before it runs, there is no guarantee that the thread does not run when a fork is done. Meaning, if the reaper runs at fork time, the child process will not be able to get connections from the pool because the mutex is locked when the child is created.
Jan-Henrik
-
Jan-Henrik Haukeland -
Michel Verbraak