The issue starts with warnings like the follogin one
[Mar 6 09:32:04] WARNING[9013] channel.c: Exceptionally long voice queue length queuing to IAX2/hbc-hah-73
about 1-2 times per second.
After some random time in the range from half a minute to 4 minutes it changes to
[Mar 6 09:36:23] WARNING[9009] chan_iax2.c: Max retries exceeded to host 144.76.51.196 on IAX2/hbc-hah-73 (type = 6, subclass = 2, ts=377730, seqno=114)
every 10 seconds until I kill (-9) Asterisk.
In that state Asterisk still answers IAX packets so that it still qualifies OK on the calling instance.
However it completely stops processing any calls.
On the calling box all following Dial()s to the locked up one end with DIALSTATUS=CANCEL and HANGUPCAUSE=0 so even they seem to be accepted, just not processed.
This particular issue started 21 days after upgrading from 11.15.0, but occurred 3 times withing two weeks so far. So I'm unsure if it's related to the upgrade.
gdb -ex "thread apply all bt":
[New LWP 15907]
[New LWP 15756]
[New LWP 4797]
[New LWP 4780]
[New LWP 9024]
[New LWP 9023]
[New LWP 9022]
[New LWP 9021]
[New LWP 9020]
[New LWP 9019]
[New LWP 9018]
[New LWP 9017]
[New LWP 9016]
[New LWP 9015]
[New LWP 9014]
[New LWP 9013]
[New LWP 9012]
[New LWP 9011]
[New LWP 9010]
[New LWP 9009]
[New LWP 9008]
[New LWP 9007]
[New LWP 9006]
[New LWP 9005]
[New LWP 9004]
[New LWP 9003]
[New LWP 9002]
[New LWP 9001]
[New LWP 9000]
[New LWP 8999]
[New LWP 8998]
[New LWP 8997]
[New LWP 8996]
[New LWP 8995]
[New LWP 8994]
[New LWP 8993]
[New LWP 8992]
[New LWP 8991]
[New LWP 8990]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
0xb7688df1 in poll () from /lib/libc.so.6
Thread 40 (Thread 0xb6f33b40 (LWP 8990)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x86f5518, t=0x86f554c) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x86f5514) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x86f55a8) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 39 (Thread 0xb6ef7b40 (LWP 8991)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x08081a66 in listener (unused=0x0) at asterisk.c:1444
#2 0x081b5e1b in dummy_start (data=0x86f95b0) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 38 (Thread 0xb6ebbb40 (LWP 8992)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x81f67b4 "logger.c", lineno=1199, func=0x81f7717 <__PRETTY_FUNCTION__.17449> "logger_thread", cond_name=0x81f72a5 "&logcond", mutex_name=0x81f7297 "&logmsgs.lock", cond=0x82613a0 <logcond>, t=0x8249488 <logmsgs+8>) at lock.c:598
#3 0x08132cac in logger_thread (data=0x0) at logger.c:1199
#4 0x081b5e1b in dummy_start (data=0x86f96b8) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 37 (Thread 0xb6e7fb40 (LWP 8993)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x080816a8 in netconsole (vconsole=0x82798e0 <consoles>) at asterisk.c:1365
#2 0x081b5e1b in dummy_start (data=0xb6b00df8) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 36 (Thread 0xb6e43b40 (LWP 8994)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x81e8034 "db.c", lineno=967, func=0x81e9057 <__PRETTY_FUNCTION__.13780> "db_sync_thread", cond_name=0x81e8f38 "&dbcond", mutex_name=0x81e8299 "&dblock", cond=0x8260760 <dbcond>, t=0x8246940 <dblock>) at lock.c:598
#3 0x080e6cb6 in db_sync_thread (data=0x0) at db.c:967
#4 0x081b5e1b in dummy_start (data=0x8948158) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 35 (Thread 0xb6affb40 (LWP 8995)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8710d40, t=0x8710d74) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8710d3c) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8948158) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 34 (Thread 0xb6ac3b40 (LWP 8996)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x081b6172 in ast_wait_for_input (fd=10, ms=5000) at utils.c:1312
#2 0x081a5b20 in ast_tcptls_server_root (data=0x8249b40 <ami_desc>) at tcptls.c:704
#3 0x081b5e1b in dummy_start (data=0x8948388) at utils.c:1223
#4 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#5 0xb7694cde in clone () from /lib/libc.so.6
Thread 33 (Thread 0xb6a87b40 (LWP 8997)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0x81dd634 "channel.c", lineno=1475, func=0x81e2651 <__PRETTY_FUNCTION__.15617> "ast_channel_by_name_cb", mutex_name=0x81de569 "chan", t=0xb4ec5f70) at lock.c:313
#4 0x0808a442 in __ao2_lock (user_data=0xb4ec5fa4, lock_how=AO2_LOCK_REQ_MUTEX, file=0x81dd634 "channel.c", func=0x81e2651 <__PRETTY_FUNCTION__.15617> "ast_channel_by_name_cb", line=1475, var=0x81de569 "chan") at astobj2.c:195
#5 0x080af054 in ast_channel_by_name_cb (obj=0xb4ec5fa4, arg=0xb6a87168, data=0xb6a87154, flags=0) at channel.c:1475
#6 0x0808bbda in internal_ao2_callback (c=0x89fa2f4, flags=(unknown: 0), cb_fn=0x80aefbb <ast_channel_by_name_cb>, arg=0xb6a87168, data=0xb6a87154, type=WITH_DATA, tag=0x0, file=0x0, line=0, func=0x0) at astobj2.c:1105
#7 0x0808c0a0 in __ao2_callback_data (c=0x89fa2f4, flags=(unknown: 0), cb_fn=0x80aefbb <ast_channel_by_name_cb>, arg=0xb6a87168, data=0xb6a87154) at astobj2.c:1225
#8 0x080aefb9 in ast_channel_callback (cb_fn=0x80aefbb <ast_channel_by_name_cb>, arg=0xb6a87168, data=0xb6a87154, ao2_flags=0) at channel.c:1460
#9 0x080af5ce in ast_channel_get_by_name_prefix (name=0xb6a87168 "lcr/909-", name_len=8) at channel.c:1618
#10 0x080e7271 in ast_parse_device_state (device=0xb4f46308 "lcr/909") at devicestate.c:279
#11 0x080e7543 in _ast_device_state (device=0xb4f46308 "lcr/909", check_cache=0) at devicestate.c:350
#12 0x080e7a0b in do_state_change (device=0xb4f46308 "lcr/909", cachable=1) at devicestate.c:465
#13 0x080e7da0 in do_devstate_changes (data=0x0) at devicestate.c:557
#14 0x081b5e1b in dummy_start (data=0x8a138e0) at utils.c:1223
#15 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#16 0xb7694cde in clone () from /lib/libc.so.6
Thread 32 (Thread 0xb6a4bb40 (LWP 8998)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x8204414 "presencestate.c", lineno=299, func=0x8204643 <__PRETTY_FUNCTION__.13037> "do_presence_changes", cond_name=0x820454c "&change_pending", mutex_name=0x820455c "&state_changes.lock", cond=0x8261500 <change_pending>, t=0x824b3a8 <state_changes+8>) at lock.c:598
#3 0x08172239 in do_presence_changes (data=0x0) at presencestate.c:299
#4 0x081b5e1b in dummy_start (data=0x8948388) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 31 (Thread 0xb6a0fb40 (LWP 8999)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8a16988, t=0x8a169bc) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8a16984) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8a16840) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 30 (Thread 0xb69d3b40 (LWP 9000)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x08105d67 in do_parking_thread (ignore=0x0) at features.c:5330
#2 0x081b5e1b in dummy_start (data=0x8948388) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 29 (Thread 0xb6997b40 (LWP 9001)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8a28ed8, t=0x8a28f0c) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8a28ed4) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8a1c588) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 28 (Thread 0xb695bb40 (LWP 9002)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x8208838 "sched.c", lineno=120, func=0x8208b75 <__PRETTY_FUNCTION__.11094> "sched_run", cond_name=0x820884b "&con->sched_thread->cond", mutex_name=0x8208840 "&con->lock", cond=0x8a34534, t=0x8a29040) at lock.c:598
#3 0x08198889 in sched_run (data=0x8a29040) at sched.c:120
#4 0x081b5e1b in dummy_start (data=0x8a34570) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 27 (Thread 0xb68b8b40 (LWP 9003)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0xb68e0983 in do_pktccops (data=0x0) at res_pktccops.c:795
#2 0x081b5e1b in dummy_start (data=0x8a56310) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 26 (Thread 0xb5e73b40 (LWP 9004)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8d344a8, t=0x8d344dc) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8d344a4) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8d34540) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 25 (Thread 0xb5e37b40 (LWP 9005)):
#0 0xb71016cb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a2654 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812ea40 in __ast_cond_timedwait (filename=0x8208838 "sched.c", lineno=126, func=0x8208b75 <__PRETTY_FUNCTION__.11094> "sched_run", cond_name=0x820884b "&con->sched_thread->cond", mutex_name=0x8208840 "&con->lock", cond=0x8d34b94, t=0x8d34650, abstime=0xb5e372b0) at lock.c:666
#3 0x08198952 in sched_run (data=0x8d34650) at sched.c:126
#4 0x081b5e1b in dummy_start (data=0x8d34540) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 24 (Thread 0xb5de6b40 (LWP 9006)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d54b8c, t=0x8d54b6c) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d53ad8) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 23 (Thread 0xb5daab40 (LWP 9007)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d5690c, t=0x8d568ec) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d55858) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 22 (Thread 0xb5d6eb40 (LWP 9008)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0x81d5f70 "astobj2.c", lineno=905, func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", mutex_name=0x81d6176 "c", t=0x89fa2c0) at lock.c:313
#4 0x0808a442 in __ao2_lock (user_data=0x89fa2f4, lock_how=AO2_LOCK_REQ_WRLOCK, file=0x81d5f70 "astobj2.c", func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", line=905, var=0x81d6176 "c") at astobj2.c:195
#5 0x0808b647 in internal_ao2_link (c=0x89fa2f4, user_data=0xb4ec80ec, flags=0, tag=0x0, file=0x81d5f70 "astobj2.c", line=937, func=0x81d6292 <__PRETTY_FUNCTION__.7826> "__ao2_link") at astobj2.c:905
#6 0x0808b809 in __ao2_link (c=0x89fa2f4, obj_new=0xb4ec80ec, flags=0) at astobj2.c:937
#7 0x080ae15f in __ast_channel_alloc_ap (needqueue=1, state=4, cid_num=0xb4df793a "", cid_name=0xb4df794a "", acctcode=0x8277e2e <__ast_string_field_empty_buffer+2> "", exten=0xb4df7928 "", context=0xb4df796a "", linkedid=0x0, amaflag=0, file=0xb5fef5b4 "chan_iax2.c", line=5925, function=0xb5ff71f0 <__FUNCTION__.17403> "ast_iax2_new", name_fmt=0xb5ff175d "IAX2/%s-%d", ap=0xb5d66e74 "\036yߴW\032") at channel.c:1136
#8 0x080ae363 in __ast_channel_alloc (needqueue=1, state=4, cid_num=0xb4df793a "", cid_name=0xb4df794a "", acctcode=0x8277e2e <__ast_string_field_empty_buffer+2> "", exten=0xb4df7928 "", context=0xb4df796a "", linkedid=0x0, amaflag=0, file=0xb5fef5b4 "chan_iax2.c", line=5925, function=0xb5ff71f0 <__FUNCTION__.17403> "ast_iax2_new", name_fmt=0xb5ff175d "IAX2/%s-%d") at channel.c:1190
#9 0xb5fb3c4d in ast_iax2_new (callno=6743, state=4, capability=4096, linkedid=0x0, cachable=1) at chan_iax2.c:5925
#10 0xb5fd1914 in socket_process_helper (thread=0x8d58058) at chan_iax2.c:11469
#11 0xb5fd61dc in socket_process (thread=0x8d58058) at chan_iax2.c:12030
#12 0xb5fd692f in iax2_process_thread (data=0x8d58058) at chan_iax2.c:12141
#13 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#14 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#15 0xb7694cde in clone () from /lib/libc.so.6
Thread 21 (Thread 0xb5d32b40 (LWP 9009)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d5a97c, t=0x8d5a95c) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d598c8) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 20 (Thread 0xb5cf6b40 (LWP 9010)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0x81dd634 "channel.c", lineno=2732, func=0x81e28b2 <__PRETTY_FUNCTION__.16084> "ast_set_hangupsource", mutex_name=0x81dec0b "bridge", t=0xb4ec5f70) at lock.c:313
#4 0x0808a442 in __ao2_lock (user_data=0xb4ec5fa4, lock_how=AO2_LOCK_REQ_MUTEX, file=0x81dd634 "channel.c", func=0x81e28b2 <__PRETTY_FUNCTION__.16084> "ast_set_hangupsource", line=2732, var=0x81dec0b "bridge") at astobj2.c:195
#5 0x080b1cc3 in ast_set_hangupsource (chan=0x90eaed4, source=0xb5ceef30 "IAX2/hbc-hah-73", force=0) at channel.c:2732
#6 0xb5fc82cf in set_hangup_source_and_cause (callno=73, causecode=16 '\020') at chan_iax2.c:10139
#7 0xb5fcdf1f in socket_process_helper (thread=0x8d5b138) at chan_iax2.c:11043
#8 0xb5fd61dc in socket_process (thread=0x8d5b138) at chan_iax2.c:12030
#9 0xb5fd692f in iax2_process_thread (data=0x8d5b138) at chan_iax2.c:12141
#10 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#11 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#12 0xb7694cde in clone () from /lib/libc.so.6
Thread 19 (Thread 0xb5cbab40 (LWP 9011)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d5da5c, t=0x8d5da3c) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d5c9a8) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 18 (Thread 0xb5c7eb40 (LWP 9012)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d6025c, t=0x8d6023c) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d5f1a8) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 17 (Thread 0xb5c42b40 (LWP 9013)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d61acc, t=0x8d61aac) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d60a18) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 16 (Thread 0xb5c06b40 (LWP 9014)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0x81d5f70 "astobj2.c", lineno=905, func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", mutex_name=0x81d6176 "c", t=0x89fa2c0) at lock.c:313
#4 0x0808a442 in __ao2_lock (user_data=0x89fa2f4, lock_how=AO2_LOCK_REQ_WRLOCK, file=0x81d5f70 "astobj2.c", func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", line=905, var=0x81d6176 "c") at astobj2.c:195
#5 0x0808b647 in internal_ao2_link (c=0x89fa2f4, user_data=0xb4c0ad14, flags=0, tag=0x0, file=0x81d5f70 "astobj2.c", line=937, func=0x81d6292 <__PRETTY_FUNCTION__.7826> "__ao2_link") at astobj2.c:905
#6 0x0808b809 in __ao2_link (c=0x89fa2f4, obj_new=0xb4c0ad14, flags=0) at astobj2.c:937
#7 0x080ae15f in __ast_channel_alloc_ap (needqueue=1, state=4, cid_num=0x90efd62 "", cid_name=0x90efd72 "", acctcode=0x8277e2e <__ast_string_field_empty_buffer+2> "", exten=0x90efd50 "", context=0x90efd92 "", linkedid=0x0, amaflag=0, file=0xb5fef5b4 "chan_iax2.c", line=5925, function=0xb5ff71f0 <__FUNCTION__.17403> "ast_iax2_new", name_fmt=0xb5ff175d "IAX2/%s-%d", ap=0xb5bfee74 "F�\016\t$\n") at channel.c:1136
#8 0x080ae363 in __ast_channel_alloc (needqueue=1, state=4, cid_num=0x90efd62 "", cid_name=0x90efd72 "", acctcode=0x8277e2e <__ast_string_field_empty_buffer+2> "", exten=0x90efd50 "", context=0x90efd92 "", linkedid=0x0, amaflag=0, file=0xb5fef5b4 "chan_iax2.c", line=5925, function=0xb5ff71f0 <__FUNCTION__.17403> "ast_iax2_new", name_fmt=0xb5ff175d "IAX2/%s-%d") at channel.c:1190
#9 0xb5fb3c4d in ast_iax2_new (callno=2596, state=4, capability=4096, linkedid=0x0, cachable=1) at chan_iax2.c:5925
#10 0xb5fd1914 in socket_process_helper (thread=0x8d62288) at chan_iax2.c:11469
#11 0xb5fd61dc in socket_process (thread=0x8d62288) at chan_iax2.c:12030
#12 0xb5fd692f in iax2_process_thread (data=0x8d62288) at chan_iax2.c:12141
#13 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#14 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#15 0xb7694cde in clone () from /lib/libc.so.6
Thread 15 (Thread 0xb5bcab40 (LWP 9015)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0xb5fef5b4 "chan_iax2.c", lineno=12122, func=0xb5ff7696 <__PRETTY_FUNCTION__.19558> "iax2_process_thread", cond_name=0xb5fef95d "&thread->cond", mutex_name=0xb5fef94f "&thread->lock", cond=0x8d64bac, t=0x8d64b8c) at lock.c:598
#3 0xb5fd68a1 in iax2_process_thread (data=0x8d63af8) at chan_iax2.c:12122
#4 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 14 (Thread 0xb5b8eb40 (LWP 9016)):
#0 0xb6f89ee9 in ?? () from /usr/lib/libgcc_s.so.1
#1 0xb6f8a117 in ?? () from /usr/lib/libgcc_s.so.1
#2 0xb6f8b253 in _Unwind_Backtrace () from /usr/lib/libgcc_s.so.1
#3 0xb76a32d5 in backtrace () from /lib/libc.so.6
#4 0x081340d5 in ast_bt_get_addresses (bt=0xb5b8def8) at logger.c:1653
#5 0x0812d702 in __ast_pthread_mutex_lock (filename=0x81d5f70 "astobj2.c", lineno=1093, func=0x81d629d <__PRETTY_FUNCTION__.7878> "internal_ao2_callback", mutex_name=0x81d6176 "c", t=0x8a8c8e0) at lock.c:256
#6 0x0808a442 in __ao2_lock (user_data=0x8a8c914, lock_how=AO2_LOCK_REQ_RDLOCK, file=0x81d5f70 "astobj2.c", func=0x81d629d <__PRETTY_FUNCTION__.7878> "internal_ao2_callback", line=1093, var=0x81d6176 "c") at astobj2.c:195
#7 0x0808bb45 in internal_ao2_callback (c=0x8a8c914, flags=OBJ_POINTER, cb_fn=0xb650baee <timerfd_timer_cmp>, arg=0xb5b8e140, data=0x0, type=DEFAULT, tag=0x0, file=0x0, line=0, func=0x0) at astobj2.c:1093
#8 0x0808bff6 in __ao2_callback (c=0x8a8c914, flags=OBJ_POINTER, cb_fn=0xb650baee <timerfd_timer_cmp>, arg=0xb5b8e140) at astobj2.c:1212
#9 0x0808c11e in __ao2_find (c=0x8a8c914, arg=0xb5b8e140, flags=OBJ_POINTER) at astobj2.c:1243
#10 0xb650bffd in timerfd_timer_ack (handle=12, quantity=1) at res_timing_timerfd.c:186
#11 0x081aa5c1 in ast_timer_ack (handle=0x8d34c20, quantity=1) at timing.c:176
#12 0xb5fc50e3 in timing_read (id=0xb6c01dc0, fd=12, events=1, cbdata=0x0) at chan_iax2.c:9397
#13 0x08127ff9 in ast_io_wait (ioc=0x8d34cd0, howlong=1000) at io.c:292
#14 0xb5fd8d73 in network_thread (ignore=0x0) at chan_iax2.c:12612
#15 0x081b5e1b in dummy_start (data=0x8d52248) at utils.c:1223
#16 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#17 0xb7694cde in clone () from /lib/libc.so.6
Thread 13 (Thread 0xb5b52b40 (LWP 9017)):
#0 0xb76553cc in nanosleep () from /lib/libc.so.6
#1 0xb65495a7 in lock_broker (unused=0x0) at func_lock.c:211
#2 0x081b5e1b in dummy_start (data=0x8a95e38) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 12 (Thread 0xb5b16b40 (LWP 9018)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x08127ee1 in ast_io_wait (ioc=0x8dcfa20, howlong=1000) at io.c:279
#2 0xb63b82f4 in network_thread (ignore=0x0) at pbx_dundi.c:2182
#3 0x081b5e1b in dummy_start (data=0x8dd2368) at utils.c:1223
#4 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#5 0xb7694cde in clone () from /lib/libc.so.6
Thread 11 (Thread 0xb5adab40 (LWP 9019)):
#0 0xb76553cc in nanosleep () from /lib/libc.so.6
#1 0xb76551bd in sleep () from /lib/libc.so.6
#2 0xb63b86d0 in process_precache (ign=0x0) at pbx_dundi.c:2260
#3 0x081b5e1b in dummy_start (data=0x8dd0cd0) at utils.c:1223
#4 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#5 0xb7694cde in clone () from /lib/libc.so.6
Thread 10 (Thread 0xb5a9eb40 (LWP 9020)):
#0 0xb76553cc in nanosleep () from /lib/libc.so.6
#1 0xb76551bd in sleep () from /lib/libc.so.6
#2 0xb63b8528 in process_clearcache (ignore=0x0) at pbx_dundi.c:2223
#3 0x081b5e1b in dummy_start (data=0x8dd2368) at utils.c:1223
#4 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#5 0xb7694cde in clone () from /lib/libc.so.6
Thread 9 (Thread 0xb5a62b40 (LWP 9021)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0xb619b60f in scan_thread (unused=0x0) at pbx_spool.c:711
#2 0x081b5e1b in dummy_start (data=0x8deaa70) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 8 (Thread 0xb5a26b40 (LWP 9022)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0xb64fdc9f "chan_lcr.c", lineno=2050, func=0xb6500701 <__PRETTY_FUNCTION__.14993> "lock_chan", mutex_name=0xb64fdc94 "&chan_lock", t=0xb6503ae0 <chan_lock>) at lock.c:313
#4 0xb64f13b3 in lock_chan () from /usr/lib/asterisk/modules/chan_lcr.so
#5 0xb64fd38a in select_main () from /usr/lib/asterisk/modules/chan_lcr.so
#6 0xb64f8ae5 in chan_thread () from /usr/lib/asterisk/modules/chan_lcr.so
#7 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#8 0xb7694cde in clone () from /lib/libc.so.6
Thread 7 (Thread 0xb5226b40 (LWP 9023)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8e11410, t=0x8e11444) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8e1140c) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8e11480) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 6 (Thread 0xb51eab40 (LWP 9024)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x820a7f8 "taskprocessor.c", lineno=313, func=0x820ae4d <__PRETTY_FUNCTION__.7739> "tps_processing_function", cond_name=0x820ab49 "&i->poll_cond", mutex_name=0x820ab32 "&i->taskprocessor_lock", cond=0x8e6e568, t=0x8e6e59c) at lock.c:598
#3 0x081a37f7 in tps_processing_function (data=0x8e6e564) at taskprocessor.c:313
#4 0x081b5e1b in dummy_start (data=0x8e4e0c8) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 5 (Thread 0xb5172b40 (LWP 4780)):
#0 0xb7101321 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0xb76a25fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#2 0x0812e5cc in __ast_cond_wait (filename=0x81d68b8 "autoservice.c", lineno=103, func=0x81d697c <__PRETTY_FUNCTION__.12573> "autoservice_run", cond_name=0x81d68e4 "&as_cond", mutex_name=0x81d68d7 "&aslist.lock", cond=0x8260460 <as_cond>, t=0x8245508 <aslist+8>) at lock.c:598
#3 0x080908e6 in autoservice_run (ign=0x0) at autoservice.c:103
#4 0x081b5e1b in dummy_start (data=0xb6b2fc70) at utils.c:1223
#5 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#6 0xb7694cde in clone () from /lib/libc.so.6
Thread 4 (Thread 0xb5136b40 (LWP 4797)):
#0 0xb7682ff1 in read () from /lib/libc.so.6
#1 0x0819c909 in inotify_daemon (data=0x0) at stdtime/localtime.c:310
#2 0x081b5e1b in dummy_start (data=0xb6b38ca0) at utils.c:1223
#3 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#4 0xb7694cde in clone () from /lib/libc.so.6
Thread 3 (Thread 0xb50beb40 (LWP 15756)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0xb64fdc9f "chan_lcr.c", lineno=2846, func=0xb65008db <__PRETTY_FUNCTION__.15131> "lcr_write", mutex_name=0xb64fdc94 "&chan_lock", t=0xb6503ae0 <chan_lock>) at lock.c:313
#4 0xb64f1ae9 in lcr_write () from /usr/lib/asterisk/modules/chan_lcr.so
#5 0x080b8f67 in ast_write (chan=0xb4ec5fa4, fr=0xb4e4ed18) at channel.c:5251
#6 0x080c04b0 in ast_generic_bridge (c0=0x90eaed4, c1=0xb4ec5fa4, config=0xb50b8544, fo=0xb50b6a2c, rc=0xb50b6a28) at channel.c:7709
#7 0x080c23db in ast_channel_bridge (c0=0x90eaed4, c1=0xb4ec5fa4, config=0xb50b8544, fo=0xb50b6a2c, rc=0xb50b6a28) at channel.c:8105
#8 0x081031e6 in ast_bridge_call (chan=0x90eaed4, peer=0xb4ec5fa4, config=0xb50b8544) at features.c:4489
#9 0xb5f72b37 in try_calling (qe=0xb50b8a8c, opts=..., opt_args=0xb50b8a84, announceoverride=0x0, url=0x0, tries=0xb50b9c0c, noption=0xb50b9c08, agi=0x0, macro=0x0, gosub=0x0, ringing=0) at app_queue.c:5974
#10 0xb5f76f26 in queue_exec (chan=0x90eaed4, data=0xb50bbdf8 "warteschlange") at app_queue.c:7215
#11 0x081523b9 in pbx_exec (c=0x90eaed4, app=0x8e5b580, data=0xb50bbdf8 "warteschlange") at pbx.c:1622
#12 0x0815b69a in pbx_extension_helper (c=0x90eaed4, con=0x0, context=0x90eba88 "ivr-main", exten=0x90ebad8 "s", priority=20, label=0x0, callerid=0x8acd9e0 "########444", action=E_SPAWN, found=0xb50be220, combined_find_spawn=1) at pbx.c:4915
#13 0x0815e4f2 in ast_spawn_extension (c=0x90eaed4, context=0x90eba88 "ivr-main", exten=0x90ebad8 "s", priority=20, callerid=0x8acd9e0 "########444", found=0xb50be220, combined_find_spawn=1) at pbx.c:6037
#14 0x0815fa6b in __ast_pbx_run (c=0x90eaed4, args=0x0) at pbx.c:6512
#15 0x08161116 in pbx_thread (data=0x90eaed4) at pbx.c:6842
#16 0x081b5e1b in dummy_start (data=0x8b3a270) at utils.c:1223
#17 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#18 0xb7694cde in clone () from /lib/libc.so.6
Thread 2 (Thread 0xb5082b40 (LWP 15907)):
#0 0xb7103c5d in __lll_lock_wait () from /lib/libpthread.so.0callerid
#1 0xb70ff02d in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0xb76a2804 in pthread_mutex_lock () from /lib/libc.so.6
#3 0x0812d7b5 in __ast_pthread_mutex_lock (filename=0x81d5f70 "astobj2.c", lineno=905, func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", mutex_name=0x81d6176 "c", t=0x89fa2c0) at lock.c:313
#4 0x0808a442 in __ao2_lock (user_data=0x89fa2f4, lock_how=AO2_LOCK_REQ_WRLOCK, file=0x81d5f70 "astobj2.c", func=0x81d6280 <__PRETTY_FUNCTION__.7811> "internal_ao2_link", line=905, var=0x81d6176 "c") at astobj2.c:195
#5 0x0808b647 in internal_ao2_link (c=0x89fa2f4, user_data=0xb4ecb154, flags=0, tag=0x0, file=0x81d5f70 "astobj2.c", line=937, func=0x81d6292 <__PRETTY_FUNCTION__.7826> "__ao2_link") at astobj2.c:905
#6 0x0808b809 in __ao2_link (c=0x89fa2f4, obj_new=0xb4ecb154, flags=0) at astobj2.c:937
#7 0x080ae15f in __ast_channel_alloc_ap (needqueue=1, state=1, cid_num=0x0, cid_name=0x0, acctcode=0x0, exten=0x0, context=0x0, linkedid=0x0, amaflag=0, file=0xb64fdc9f "chan_lcr.c", line=2129, function=0xb650082d <__FUNCTION__.15021> "lcr_request", name_fmt=0xb64fdec6 "%s/%d", ap=0xb507c434 "�4P�\216\003") at channel.c:1136
#8 0x080ae363 in __ast_channel_alloc (needqueue=1, state=1, cid_num=0x0, cid_name=0x0, acctcode=0x0, exten=0x0, context=0x0, linkedid=0x0, amaflag=0, file=0xb64fdc9f "chan_lcr.c", line=2129, function=0xb650082d <__FUNCTION__.15021> "lcr_request", name_fmt=0xb64fdec6 "%s/%d") at channel.c:1190
#9 0xb64f5bd2 in lcr_request () from /usr/lib/asterisk/modules/chan_lcr.so
#10 0x080bafbb in ast_request (type=0xb6cd1784 "lcr", request_cap=0xb6d56f60, requestor=0xb4bfac8c, addr=0xb6cd178c "hipath/24499", cause=0xb507da58) at channel.c:5973
#11 0xb63fce9a in dial_exec_full (chan=0xb4bfac8c, data=0xb507fdf8 "lcr/hipath/24499,16,U(queuelog,s,1)", peerflags=0xb507dc48, continue_exec=0x0) at app_dial.c:2434
#12 0xb6400dd1 in dial_exec (chan=0xb4bfac8c, data=0xb507fdf8 "lcr/hipath/24499,16,U(queuelog,s,1)") at app_dial.c:3130
#13 0x081523b9 in pbx_exec (c=0xb4bfac8c, app=0x8dd9a00, data=0xb507fdf8 "lcr/hipath/24499,16,U(queuelog,s,1)") at pbx.c:1622
#14 0x0815b69a in pbx_extension_helper (c=0xb4bfac8c, con=0x0, context=0xb4bfb840 "ivr-main", exten=0xb4bfb890 "s", priority=10, label=0x0, callerid=0xb4f5ee08 "########45", action=E_SPAWN, found=0xb5082220, combined_find_spawn=1) at pbx.c:4915
#15 0x0815e4f2 in ast_spawn_extension (c=0xb4bfac8c, context=0xb4bfb840 "ivr-main", exten=0xb4bfb890 "s", priority=10, callerid=0xb4f5ee08 "########45", found=0xb5082220, combined_find_spawn=1) at pbx.c:6037
#16 0x0815fa6b in __ast_pbx_run (c=0xb4bfac8c, args=0x0) at pbx.c:6512
#17 0x08161116 in pbx_thread (data=0xb4bfac8c) at pbx.c:6842
#18 0x081b5e1b in dummy_start (data=0xb4b64a28) at utils.c:1223
#19 0xb70fd087 in start_thread () from /lib/libpthread.so.0
#20 0xb7694cde in clone () from /lib/libc.so.6
Thread 1 (Thread 0xb6f92700 (LWP 17668)):
#0 0xb7688df1 in poll () from /lib/libc.so.6
#1 0x080877d0 in monitor_sig_flags (unused=0x0) at asterisk.c:3528
#2 0x0808a1bc in main (argc=2, argv=0xbfeaa0d4) at asterisk.c:4428
asterisk -x "core show locks":
=======================================================================
=== 11.16.0
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0xb6a87b40 (do_devstate_changes started at [ 750] devicestate.c ast_device_state_engine_init())
=== ---> Lock #0 (astobj2.c): MUTEX 1093 internal_ao2_callback c 0x89fa2c0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/astobj2.c:1097 internal_ao2_callback()
main/astobj2.c:1226 __ao2_callback_data() (0x808c04b+55)
main/channel.c:1461 ast_channel_callback() (0x80aef8a+2F)
main/channel.c:1618 ast_channel_get_by_name_prefix() (0x80af591+3D)
main/devicestate.c:279 ast_parse_device_state() (0x80e722f+42)
main/devicestate.c:350 _ast_device_state()
main/devicestate.c:465 do_state_change()
main/devicestate.c:558 do_devstate_changes()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== ---> Waiting for Lock #1 (channel.c): MUTEX 1475 ast_channel_by_name_cb chan 0xb4ec5f70 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/channel.c:2733 ast_set_hangupsource() (0x80b1bcc+F7)
channels/chan_iax2.c:10140 set_hangup_source_and_cause()
channels/chan_iax2.c:11044 socket_process_helper()
channels/chan_iax2.c:12030 socket_process()
channels/chan_iax2.c:12142 iax2_process_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: channel.c line 4987 (ast_write)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb5d6eb40 (iax2_process_thread started at [12637] chan_iax2.c start_network_thread())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 905 internal_ao2_link c 0x89fa2c0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/astobj2.c:906 internal_ao2_link()
main/astobj2.c:938 __ao2_link() (0x808b7ca+3F)
main/channel.c:1144 __ast_channel_alloc_ap()
main/channel.c:1190 __ast_channel_alloc() (0x80ae2f1+72)
channels/chan_iax2.c:5925 ast_iax2_new()
channels/chan_iax2.c:11469 socket_process_helper()
channels/chan_iax2.c:12030 socket_process()
channels/chan_iax2.c:12142 iax2_process_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: astobj2.c line 1093 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb5cf6b40 (iax2_process_thread started at [12637] chan_iax2.c start_network_thread())
=== ---> Waiting for Lock #0 (channel.c): MUTEX 2732 ast_set_hangupsource bridge 0xb4ec5f70 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/channel.c:2733 ast_set_hangupsource() (0x80b1bcc+F7)
channels/chan_iax2.c:10140 set_hangup_source_and_cause()
channels/chan_iax2.c:11044 socket_process_helper()
channels/chan_iax2.c:12030 socket_process()
channels/chan_iax2.c:12142 iax2_process_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: channel.c line 4987 (ast_write)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb5c06b40 (iax2_process_thread started at [12637] chan_iax2.c start_network_thread())
=== ---> Waiting for Lock #0 (astobj2.c): MUTEX 905 internal_ao2_link c 0x89fa2c0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/astobj2.c:906 internal_ao2_link()
main/astobj2.c:938 __ao2_link() (0x808b7ca+3F)
main/channel.c:1144 __ast_channel_alloc_ap()
main/channel.c:1190 __ast_channel_alloc() (0x80ae2f1+72)
channels/chan_iax2.c:5925 ast_iax2_new()
channels/chan_iax2.c:11469 socket_process_helper()
channels/chan_iax2.c:12030 socket_process()
channels/chan_iax2.c:12142 iax2_process_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: astobj2.c line 1093 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb50beb40 (pbx_thread started at [ 6868] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 4987 ast_write chan 0xb4ec5f70 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:380 __ast_pthread_mutex_trylock() (0x812d9e6+83)
main/astobj2.c:301 __ao2_trylock() (0x808a66d+7C)
main/channel.c:4987 ast_write() (0x80b7c43+EE)
main/channel.c:7712 ast_generic_bridge()
main/channel.c:8105 ast_channel_bridge() (0x80c0d73+1668)
main/features.c:4489 ast_bridge_call() (0x8102566+C80)
apps/app_queue.c:5974 try_calling()
apps/app_queue.c:7215 queue_exec()
main/pbx.c:1622 pbx_exec() (0x81521f0+1C9)
main/pbx.c:4915 pbx_extension_helper()
main/pbx.c:6038 ast_spawn_extension() (0x815e49f+53)
main/pbx.c:6512 __ast_pbx_run()
main/pbx.c:6843 pbx_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== ---> Waiting for Lock #1 (chan_lcr.c): MUTEX 2846 lcr_write &chan_lock 0xb6503ae0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
:0 lock_chan() (0xb64f1370+43)
:0 select_main() (0xb64fd030+35A)
chan_lcr.c:0 chan_thread()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: chan_lcr.c line 2107 (lcr_request)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb5082b40 (pbx_thread started at [ 6868] pbx.c ast_pbx_start())
=== ---> Lock #0 (chan_lcr.c): MUTEX 2107 lcr_request &chan_lock 0xb6503ae0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
chan_lcr.c:0 lcr_request()
main/channel.c:5973 ast_request() (0x80bad21+29A)
apps/app_dial.c:2434 dial_exec_full()
apps/app_dial.c:3131 dial_exec()
main/pbx.c:1622 pbx_exec() (0x81521f0+1C9)
main/pbx.c:4915 pbx_extension_helper()
main/pbx.c:6038 ast_spawn_extension() (0x815e49f+53)
main/pbx.c:6512 __ast_pbx_run()
main/pbx.c:6843 pbx_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== ---> Waiting for Lock #1 (astobj2.c): MUTEX 905 internal_ao2_link c 0x89fa2c0 (1)
main/logger.c:1653 ast_bt_get_addresses() (0x81340bc+19)
main/lock.c:258 __ast_pthread_mutex_lock() (0x812d67f+83)
main/astobj2.c:195 __ao2_lock() (0x808a3c6+7C)
main/astobj2.c:906 internal_ao2_link()
main/astobj2.c:938 __ao2_link() (0x808b7ca+3F)
main/channel.c:1144 __ast_channel_alloc_ap()
main/channel.c:1190 __ast_channel_alloc() (0x80ae2f1+72)
channels/chan_iax2.c:5925 ast_iax2_new()
channels/chan_iax2.c:11469 socket_process_helper()
channels/chan_iax2.c:12030 socket_process()
channels/chan_iax2.c:12142 iax2_process_thread()
main/utils.c:1223 dummy_start()
:0 start_thread()
:0 __clone() (0xb7694c80+5E)
=== --- ---> Locked Here: astobj2.c line 1093 (internal_ao2_callback)
=== -------------------------------------------------------------------
===
=======================================================================
asterisk -x "core show threads":
0xb51aeb40 19626 netconsole started at [ 1486] asterisk.c listener()
0xb5082b40 15907 pbx_thread started at [ 6868] pbx.c ast_pbx_start()
0xb50beb40 15756 pbx_thread started at [ 6868] pbx.c ast_pbx_start()
0xb5136b40 4797 inotify_daemon started at [ 349] stdtime/localtime.c add_notify()
0xb5172b40 4780 autoservice_run started at [ 228] autoservice.c ast_autoservice_start()
0xb51eab40 9024 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb5226b40 9023 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb5a62b40 9021 scan_thread started at [ 902] pbx_spool.c load_module()
0xb5a9eb40 9020 process_clearcache started at [ 2272] pbx_dundi.c start_network_thread()
0xb5adab40 9019 process_precache started at [ 2271] pbx_dundi.c start_network_thread()
0xb5b16b40 9018 network_thread started at [ 2270] pbx_dundi.c start_network_thread()
0xb5b52b40 9017 lock_broker started at [ 524] func_lock.c load_module()
0xb5b8eb40 9016 network_thread started at [12659] chan_iax2.c start_network_thread()
0xb5bcab40 9015 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5c06b40 9014 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5c42b40 9013 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5c7eb40 9012 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5cbab40 9011 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5cf6b40 9010 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5d32b40 9009 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5d6eb40 9008 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5daab40 9007 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5de6b40 9006 iax2_process_thread started at [12637] chan_iax2.c start_network_thread()
0xb5e37b40 9005 sched_run started at [ 182] sched.c ast_sched_start_thread()
0xb5e73b40 9004 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb68b8b40 9003 do_pktccops started at [ 1015] res_pktccops.c restart_pktc_thread()
0xb695bb40 9002 sched_run started at [ 182] sched.c ast_sched_start_thread()
0xb6997b40 9001 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb69d3b40 9000 do_parking_thread started at [ 9090] features.c ast_features_init()
0xb6a0fb40 8999 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb6a87b40 8997 do_devstate_changes started at [ 750] devicestate.c ast_device_state_engine_init()
0xb6a4bb40 8998 do_presence_changes started at [ 318] presencestate.c ast_presence_state_engine_init()
0xb6ac3b40 8996 desc->accept_fn started at [ 1019] tcptls.c ast_tcptls_server_start()
0xb6affb40 8995 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
0xb6e43b40 8994 db_sync_thread started at [ 1018] db.c astdb_init()
0xb6e7fb40 8993 netconsole started at [ 1486] asterisk.c listener()
0xb6ebbb40 8992 logger_thread started at [ 1294] logger.c init_logger()
0xb6ef7b40 8991 listener started at [ 1546] asterisk.c ast_makesocket()
0xb6f33b40 8990 tps_processing_function started at [ 471] taskprocessor.c ast_taskprocessor_get()
39 threads listed.
asterisk -x "core show taskprocessors":
+----- Processor -----+--- Processed ---+- In Queue -+- Max Depth -+
app_queue 2785 0 1
core_event_dispatcher 2794 0 2
app_voicemail 0 0 0
pbx-core 2787 0 1
ast_msg_queue 0 0 0
CCSS core 0 0 0
iax2_transmit 19583 0 1
+---------------------+-----------------+------------+-------------+
7 taskprocessors
asterisk -x "iax2 show stats":
IAX Statistics
---------------------
Outstanding frames: 3 (0 ingress, 3 egress)
280193 timed and 0 untimed transmits; MTU 1040/0/1410
Packets in transmit queue: 0 dead, 0 final, 3 total