Summary: | ASTERISK-18142: unresponsive to sip requests | ||||
Reporter: | Ole Kaas (ole.kaas) | Labels: | |||
Date Opened: | 2011-07-15 04:00:21 | Date Closed: | 2011-08-17 15:27:19 | ||
Priority: | Major | Regression? | |||
Status: | Closed/Complete | Components: | Channels/chan_sip/General | ||
Versions: | 1.8.5.0 | Frequency of Occurrence | Frequent | ||
Related Issues: |
| ||||
Environment: | Linux (Debian 6.0) 2.6.32-5-686 #1 SMP Mon Jun 13 04:13:06 UTC 2011 i686 GNU/Linux | Attachments: | ( 0) backtrace_crash_11-08-11.txt ( 1) backtrace2.txt ( 2) bt.2011-07-15T08-15-05-0200.txt ( 3) lock.2011-07-29.txt ( 4) singler_backtrace-threads_11-08-05.txt ( 5) singler_core-show-locks_11-08-05.txt ( 6) timerfd_fix.diff | ||
Description: | All of a sudden * wont respond to sip requests (examined with ngrep). It is possible to login to the consoe with 'asterisk -r' and 'core show channels' shows no active calls - wich is never the case during normal office hours. So far the fix is to 'kill -9 pidof*' and wait for safe_asterisk to bring it up again. Happens 2-3 times a day. Attached a backtrace from a core dump produced with 'kill -11' The issue seems to happen on version 1.8.4.4 too. That system, however, is only under light load - so it might happen once a week. | ||||
Comments: | By: Ole Kaas (ole.kaas) 2011-07-15 04:01:36.150-0500 backtrace By: Leif Madsen (lmadsen) 2011-07-26 09:55:28.969-0500 This is likely a problem with res_timing_timerfd, so for a work around you should use res_timing_dahdi for now. Hopefully we'll get res_timing_timerfd fixed up here shortly. By: caspy (caspy) 2011-07-27 07:48:03.371-0500 i'm getting a deadlock like this with res_timing_dahdi. next lock i'll try to suply bt. By: KC (kchan1028) 2011-07-28 12:51:44.551-0500 I am having the exact same problem all of the sudden on Asterisk 1.6.2.19 and Asterisk 1.8.4 but not on Asterisk 1.4. There is no ERROR show in the log file other than the normal ERROR "Dropping this SIP message with Call-ID". My setup is very simple and use FASTAGI. I recompiled with DEBUG_THREADS and DONT_OPTIMIZE. I tired to get the backtrace but the backtrace is not available.It say No Stack. Here is the first core show locks cat core-show-locks.txt ======================================================================= === Currently Held Locks ============================================== ======================================================================= === === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked) === === Thread ID: 140357420693248 (do_monitor started at [24746] chan_sip.c restart_monitor()) === ---> Lock #0 (chan_sip.c): MUTEX 24235 handle_request_do &netlock 0x7fa78adc9c60 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/lib/asterisk/modules/chan_sip.so(+0x7b9c1) [0x7fa78ab7e9c1] /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6] /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106] /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === ---> Lock #1 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445658] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x445b7d] /usr/sbin/asterisk(__ao2_find+0x31) [0x445cef] /usr/sbin/asterisk() [0x46bc60] /usr/sbin/asterisk(ast_channel_get_by_name+0x27) [0x46bc91] /usr/lib/asterisk/modules/func_channel.so(+0x46a6) [0x7fa7945676a6] /usr/sbin/asterisk(ast_func_write+0x15e) [0x503b87] /usr/sbin/asterisk(pbx_builtin_setvar_helper+0xba) [0x51800c] /usr/lib/asterisk/modules/chan_sip.so(+0x7a30b) [0x7fa78ab7d30b] /usr/lib/asterisk/modules/chan_sip.so(+0x7bf3f) [0x7fa78ab7ef3f] /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6] /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106] /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === ---> Waiting for Lock #2 (channel.c): MUTEX 1641 ast_channel_cmp_cb chan 0x7fa749058f50 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x46b872] /usr/sbin/asterisk() [0x445749] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x445b7d] /usr/sbin/asterisk(__ao2_find+0x31) [0x445cef] /usr/sbin/asterisk() [0x46bc60] /usr/sbin/asterisk(ast_channel_get_by_name+0x27) [0x46bc91] /usr/lib/asterisk/modules/func_channel.so(+0x46a6) [0x7fa7945676a6] /usr/sbin/asterisk(ast_func_write+0x15e) [0x503b87] /usr/sbin/asterisk(pbx_builtin_setvar_helper+0xba) [0x51800c] /usr/lib/asterisk/modules/chan_sip.so(+0x7a30b) [0x7fa78ab7d30b] /usr/lib/asterisk/modules/chan_sip.so(+0x7bf3f) [0x7fa78ab7ef3f] /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6] /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106] /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: channel.c line 3661 (__ast_read) === ------------------------------------------------------------------- === === Thread ID: 140357415556864 (netconsole started at [ 1344] asterisk.c listener()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 842 internal_ao2_iterator_next a->c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357410477824 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351656638208 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357400827648 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357402351360 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351665272576 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357405906688 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351658161920 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357386098432 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357390669568 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357404382976 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357406922496 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357403875072 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Lock #0 (channel.c): MUTEX 3661 __ast_read chan 0x7fa749058f50 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_trylock+0xd9) [0x4dd202] /usr/sbin/asterisk(__ao2_trylock+0x5a) [0x444a0a] /usr/lib/asterisk/modules/chan_sip.so(+0x7baf0) [0x7fa78ab7eaf0] /usr/lib/asterisk/modules/chan_sip.so(+0x7b7b6) [0x7fa78ab7e7b6] /usr/sbin/asterisk(ast_io_wait+0x1c4) [0x4d7106] /usr/lib/asterisk/modules/chan_sip.so(+0x7d5b2) [0x7fa78ab805b2] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === ------------------------------------------------------------------- === === Thread ID: 140357401843456 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351674414848 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351673906944 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357393209088 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351671875328 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351662733056 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357387622144 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357387114240 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351658669824 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357397272320 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357409462016 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351663748864 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357392193280 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351664256768 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140351660193536 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === === Thread ID: 140357384574720 (pbx_thread started at [ 5038] pbx.c ast_pbx_start()) === ---> Waiting for Lock #0 (astobj2.c): MUTEX 493 internal_ao2_link c 0x27c9060 (1) /usr/sbin/asterisk(ast_bt_get_addresses+0x1d) [0x4e3e40] /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xd9) [0x4dce47] /usr/sbin/asterisk(__ao2_lock+0x5a) [0x444948] /usr/sbin/asterisk() [0x445e45] /usr/sbin/asterisk(__ao2_iterator_next+0x2f) [0x446275] /usr/sbin/asterisk(ast_channel_iterator_next+0x1c) [0x46b815] /usr/sbin/asterisk() [0x487b2d] /usr/sbin/asterisk(ast_cli_command_full+0x236) [0x48c551] /usr/sbin/asterisk(ast_cli_command_multiple_full+0x97) [0x48c6ad] /usr/sbin/asterisk() [0x43c266] /usr/sbin/asterisk() [0x55dca9] /lib64/libpthread.so.0() [0x33d9006ccb] /lib64/libc.so.6(clone+0x6d) [0x33d88e0c2d] === --- ---> Locked Here: astobj2.c line 657 (internal_ao2_callback) === ------------------------------------------------------------------- === ======================================================================= By: Richard Mudgett (rmudgett) 2011-07-28 19:13:00.446-0500 KC: Your core show locks looks like the timerfd problem. Also please do not put files like a core show locks capture inline. Attach them and describe the contents. By: KC (kchan1028) 2011-07-29 03:20:51.331-0500 Richard: I apologized for that. I tried res_timing_dahdi instead timerrfd. This time asterisk didn't hang but very long SIP response time take causing PDD issue. I tested it with sipsak on the same server as asterisk. received last message 5809.219 ms after first request (test duration). biggest delay between request and response was 5809.219 ms 2 time(s) the timeout of 500 ms exceeded and request was retransmitted. SIP warning There is no backtrace again (Maybe i did something wrong. I use gbd -se asterisk <pid> | tee /tmp/backtrace.txt). I am going to attach the core show locks now. Thank You By: KC (kchan1028) 2011-07-29 03:22:21.792-0500 core show locks By: KC (kchan1028) 2011-07-29 11:49:00.823-0500 I recompiled again without DEBUG_THREADS and DONT_OPTIMIZE and it worked fine for the past few hours now. By: Ole Kaas (ole.kaas) 2011-08-08 04:34:25.978-0500 After noload => timerfd the system have been rock stable. So I've gone ahead and upgraded a much more busy server. This is turning into a nightmare - deadlock within an hour or even shorter. This system is 64bit. Again - you can connect to the console and "core show channels" is empty. Howner - some times there are calls running - but sip is unresponsive (no backtrace yet). Linux (Debian) 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC 2011 x86_64 GNU/Linux By: Ole Kaas (ole.kaas) 2011-08-08 06:57:09.322-0500 Investigating this issue further (on the 64bit system). All breakdowns seems to happen right after a reload. We have a script that checks for config changes every 10 minutes and it sends a "reload" to asterisk if it is required. All alerts so far are 0 or 1 minute behind auto reload (sipsak connect attempt every minute) By: singler (singler) 2011-08-08 07:36:03.224-0500 I am experiencing similar problem (32bit system). I am attaching my backtraces, later asterisk debug log will follow By: singler (singler) 2011-08-11 01:17:01.513-0500 After using dahdi timer I did not receive "threadlock" in two days, but today asterisk crashed. In syslog I can see: kernel: [9552804.194007] asterisk[10583]: segfault at 4 ip b57db4d2 sp b17fe230 error 4 in chan_sip.so[b5768000+b3000] I am attaching backtrace_crash_11-08-11.txt Size of backtrace was 3GB, so I suspect a memory leak (I also reported at ASTERISK-18247 issue). Could you guys check if your systems do not experience memory leak? By: Ole Kaas (ole.kaas) 2011-08-15 10:26:16.682-0500 After upgrading to 1.8.6.0-rc1 this issue seems to have vanished. We have issued several full reloads today when needed due to config changes. Not a single deadlock so far. By: Terry Wilson (twilson) 2011-08-15 13:58:46.267-0500 timerfd_fix.diff is the patch from https://reviewboard.asterisk.org/r/1361/ for 1.8 which should fix this issue. Please test! By: Ole Kaas (ole.kaas) 2011-08-15 17:46:08.155-0500 Patch applied to 1.8.5.0 on server with arch i386. Compiled with DONT_OPTIMIZE DEBUG_THREADS MALLOC_DEBUG noload => res_timer_timerfd.so commented out in config. verified module in use with "modules show" By: Terry Wilson (twilson) 2011-08-17 10:33:16.657-0500 Ole: And did it work? By: Terry Wilson (twilson) 2011-08-17 10:37:13.085-0500 Also, I'm guessing that 1.8.6.0-rc1 has an earlier version of the patch applied which was reverted because it caused some other issues. By: Ole Kaas (ole.kaas) 2011-08-17 10:48:28.671-0500 Yes. I decided to wait one more day to be sure. There have not been any deadlocks at all. The server (amd64) running 1.8.6.0-rc1 is currently using res_timing_pthread.so. However those deadlocks was caused by reloads and the timing wasn't reverted to dahdi after upgrade from 1.8.5.0. We will probably revert to dahdi or even fd, but it seems to be stable and we need some quiet time now :) By: Terry Wilson (twilson) 2011-08-17 15:28:14.407-0500 Fixed in 1.8 r332320 |