Summary:ASTERISK-07938: Out of idle IAX2 threads for I/O
Reporter:Dawid M (dmielnik)Labels:
Date Opened:2006-10-17 04:20:19Date Closed:2006-11-02 10:34:21.000-0600
Versions:Frequency of
Environment:Attachments:( 0) cli.txt
( 1) cli2.txt
( 2) info_thread.txt
( 3) log_full.tgz
( 4) log_full2.tgz
( 5) non_opt_info_thread.txt
( 6) non_opt_thread_apply_all_bt.txt
( 7) thread_apply_all_bt.txt
Description:After working for a couple of minutes on a server with small IAX load (couple of calls per minute) Asterisk runs out of idle IAX2 threads. Problem can be easily repeated - the server works for a couple of minutes then stops responding to IAX.

chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!


IAX2 Thread Information
Idle Threads:LI>
Active Threads:>
Thread D99: state=2, update=61, actions=2, func ='socket_process'
Thread D98: state=2, update=61, actions=3, func ='socket_process'
Thread D97: state=2, update=61, actions=11, func ='socket_process'
Thread P6: state=2, update=84, actions=7045, func ='socket_process'
Thread P5: state=2, update=85, actions=7017, func ='socket_process'
Thread P8: state=2, update=85, actions=7010, func ='socket_process'
Thread P1: state=2, update=86, actions=7016, func ='socket_process'
Dynamic Threads:
110 of 10 threads accounted for with 0 dynamic threads


I have also compiled asterisk with 500 (instead of 100) dynamic threads - server worked a bit longer but eventually IAX2 blocked - out of idle threads:

Thread P5: state=2, update=23935, actions=144933, func ='socket_process'
Thread P7: state=2, update=23935, actions=144933, func ='expire_registry'
Thread P6: state=2, update=23935, actions=144878, func ='expire_registry'
Thread P4: state=2, update=23935, actions=144920, func ='socket_process'
Thread P10: state=2, update=23935, actions=144770, func ='expire_registry'
Dynamic Threads:
510 of 10 threads accounted for with 0 dynamic threads

Fedora Core (2.6.5-1.358smp)
Dual Xeon (64 bit)
Comments:By: Dawid M (dmielnik) 2006-10-17 04:22:45

I seem to have put the wrong category - its not really IAX Database support

By: Joshua C. Colp (jcolp) 2006-10-17 15:50:21

Can you please use gdb to acquire a backtrace? It will allow me to see where every thread is hanging, or would it be possible to arrange access so I could try to diagnose it? Thanks.

By: Dawid M (dmielnik) 2006-10-17 17:45:54

ok - attached the gdb back traces
please contect me - dmielnik (at) voiceware.pl if you would still want to arrange access to the server.

By: Joshua C. Colp (jcolp) 2006-10-18 12:38:37

Is this an unoptimized build?

By: Dawid M (dmielnik) 2006-10-18 12:48:52

No - this is an optimized build. I am not sure but I think I have tried compiling with DONT_OPTIMIZE with the same effect.

By: Joshua C. Colp (jcolp) 2006-10-18 12:50:22

The issue may still reoccur with an unoptimized build but in order to accurately track it down from a backtrace it needs to be unoptimized.

By: Dawid M (dmielnik) 2006-10-18 12:53:44

ok - I will provide you with backtraces of the deadlock for an unoptimazed build in about 20 minutes.

By: Dawid M (dmielnik) 2006-10-18 14:00:50

ok - uploaded backtrace for non optimized build

By: Joshua C. Colp (jcolp) 2006-11-02 10:34:20.000-0600

This issue should be resolved in the latest chan_iax2 commit I did. Give it a go. If not please reopen.