ASTERISK-16692: deadlock in local_bridgedchannel during masquerade

[Home]

Summary: ASTERISK-16692: deadlock in local_bridgedchannel during masquerade

Reporter: Alec Davis (alecdavis) Labels:

Date Opened: 2010-09-17 06:07:51 Date Closed: 2011-06-03 23:36:29

Priority: Minor Regression? No

Status: Closed/Complete Components: Channels/chan_local

Versions: Frequency of
Occurrence

Related
Issues:

Environment: Attachments: ( 0) backtrace-2-mar14.txt
( 1) core-show-locks2-mar14.txt
( 2) csl-bt-merged-mar14.txt
( 3) locks.txt

Description: while testing ASTERISK-14975 I now have a very stable system, but when pushing the number of loops in a looped local channel, I finish up with a deadlock.

****** ADDITIONAL INFORMATION ******

Full 'core show locks' in attached file: locks.txt

dialplan to reliably reproduce this:
Dial 10020 for 20 loops.
[test]
exten => 10000,1,Answer()
exten => 10000,n,Playback(test-tones-follow)
exten => 10000,n,Milliwatt()

exten => _1XXXX,1,Set(i=${MATH(${EXTEN}-1,int)})
exten => _1XXXX,n,Dial(Local/${i}@test,,r)

summary of core show locks.

asterix*CLI> core show locks

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: -1359500400 (pbx_thread started at [ 4994] pbx.c ast_pbx_start())
=== ---> Tried and failed to get Lock #0 (chan_local.c): MUTEX 382 local_queue_frame &p->lock 0x947b6b0 (0)
=== -------------------------------------------------------------------
===
=== Thread ID: -1359746160 (pbx_thread started at [ 4994] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 4571 ast_write chan 0x947ee68 (1)
=== ---> Lock #1 (chan_local.c): MUTEX 538 local_write &p->lock 0x947b6b0 (1)
=== ---> Waiting for Lock #2 (chan_local.c): MUTEX 281 local_bridgedchannel &p->lock 0x9492fd0 (1)
=== --- ---> Locked Here: chan_local.c line 538 (local_write)
=== -------------------------------------------------------------------
===
=== Thread ID: -1360483440 (pbx_thread started at [ 4994] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 4571 ast_write chan 0x9496290 (1)
=== ---> Lock #1 (chan_local.c): MUTEX 538 local_write &p->lock 0x9492fd0 (1)
=== ---> Lock #2 (chan_local.c): MUTEX 459 check_bridge p->chan->_bridge 0x949e008 (2)
=== ---> Lock #3 (chan_local.c): MUTEX 461 check_bridge p->owner 0xaf9b70b0 (2)
=== ---> Waiting for Lock #4 (chan_local.c): MUTEX 281 local_bridgedchannel &p->lock 0x947b6b0 (1)
=== --- ---> Locked Here: chan_local.c line 538 (local_write)
=== -------------------------------------------------------------------
===
=== Thread ID: -1364624496 (pbx_thread started at [ 4994] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 4571 ast_write chan 0x9510a28 (1)
=== ---> Lock #1 (chan_local.c): MUTEX 382 local_queue_frame &p->lock 0x950d268 (1)
=== -------------------------------------------------------------------
===
=======================================================================

Comments: By: Stefan Schmidt (schmidts) 2010-09-22 16:19:04

could this be related to 18028 ? i see the same test exten, maybe its just the same problem showing in different output
By: Alec Davis (alecdavis) 2010-09-22 17:30:44

No, ASTERISK-16711 had not yet got itself into a dead lock, but if I'd let it go for long enough it still does.

The test dialplan came about in March ASTERISK-15819, and has proven to highlight a few problems, which have slowly been fixed.
By: Alec Davis (alecdavis) 2011-03-15 03:32:26

file chan_local.c:

static void check_bridge(struct local_pvt *p)
{
struct ast_channel_monitor *tmp;
if (ast_test_flag(p, LOCAL_ALREADY_MASQED) || ast_test_flag(p, LOCAL_NO_OPTIMIZATION) || !p->chan || !p->owner ) {
return;
}

/* This will dead lock the other channel */
if (p->chan->_bridge != ast_bridged_channel(p->chan)) {
return;
}
By: Alec Davis (alecdavis) 2011-03-15 03:58:39

Asterisk SVN-trunk-r310373M

backtrace-2-mar14
core-show-locks2-mar14

csl-bt-merged-mar14.txt: merged Core Show Locks and Backtrace of previous uploads of the 2 threads in deadlock.

invoked using looping localchan test dialplan by dialling 10032<pre>[phones]
exten => 10000,1,Goto(echo-test,s,1)

exten => _1XXXX,1,Set(i=${MATH(${EXTEN}-1,int)})
exten => _1XXXX,n,Dial(Local/${i}@phones)</pre>

By: Alec Davis (alecdavis) 2011-05-26 06:19:27

This will be fixed when https://reviewboard.asterisk.org/r/1231/ is shipped.
By: Alec Davis (alecdavis) 2011-06-03 23:36:29

fixed by https://reviewboard.asterisk.org/r/1231/