[Home]

Summary:ASTERISK-14727: [patch] chan_local deadlock
Reporter:Tim Ringenbach at Asteria Solutions Group (tim_ringenbach)Labels:
Date Opened:2009-08-27 19:03:12Date Closed:2009-08-31 11:22:53
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Channels/chan_local
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) chan_local.diff
Description:I've experienced a chan_local deadlock that wasn't in 1.4.24 but is in 1.4.25.1 and 1.4.26.1. I see threads stuck like this:

 28 process 7567  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0
 27 process 7568  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0
 26 process 7569  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0
 25 process 7570  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0
 24 process 7571  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0


And this backtrace from them:
#0  0x00007fb9de31a174 in __lll_lock_wait () from /lib/libpthread.so.0
#1  0x00007fb9de315b23 in _L_lock_261 () from /lib/libpthread.so.0
#2  0x00007fb9de3154e8 in pthread_mutex_lock () from /lib/libpthread.so.0
#3  0x00007fb9cfe1840b in local_write (ast=0x7fb9d0019cf0, f=0x80) at chan_local.c:340
#4  0x00007fb9de31f3d1 in ?? () from /lib/libpthread.so.0
ASTERISK-1  0x0000000000000001 in ?? ()
ASTERISK-2  0x000000004342d1e0 in ?? ()
ASTERISK-3  0x000000004342ce60 in ?? ()
ASTERISK-4  0x0000000000000001 in ?? ()
ASTERISK-5  0x00007fb9dd90aee3 in siglongjmp () from /lib/libc.so.6
ASTERISK-6 0x00007fb9de319ee8 in unwind_stop () from /lib/libpthread.so.0
ASTERISK-7 0x00000000004388a9 in ast_request (type=0x7fb9c80b9cf0 "Local", format=64, data=0x7fb9c80b9d40, cause=0x4342cd18) at channel.c:3381

(manually killed with sig 11 to ge trace, it never crashed on its own)

And I think I've found the problem, which I'll attach as a patch.
Comments:By: Tim Ringenbach at Asteria Solutions Group (tim_ringenbach) 2009-08-31 10:45:01

After further testing, this did not fix the deadlocks I was having. But, I think the patch is still valid, I'm just not hitting that potential deadlock.

What really fixed my problem was reverting r190286. I have a patch to chan_local that expands on the device state tracking, such that if any local channels are up for that extension, the device returns in use. I was deadlocking on the list lock because r190286 breaks the locking order.

So this bug might be invalid now since the deadlock only happens in my patched version.

But you are violating the locking order. Other parts of the code seem to rely on that locking order too. The unload_module() function and the locals_show() function look to be the only places that assume that locking order, so the code might deadlock on either of those.
By: Digium Subversion (svnbot) 2009-08-31 11:17:36

Repository: asterisk
Revision: 214940

U   branches/1.4/channels/chan_local.c

------------------------------------------------------------------------
r214940 | tilghman | 2009-08-31 11:17:35 -0500 (Mon, 31 Aug 2009) | 7 lines

Also unlock the "other" channel, when returning, due to glare.
(closes issue ASTERISK-14727)
Reported by: tim_ringenbach
Patches:
      chan_local.diff uploaded by tim ringenbach (license 540)
Tested by: tim_ringenbach

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=214940
By: Digium Subversion (svnbot) 2009-08-31 11:19:16

Repository: asterisk
Revision: 214945

_U  trunk/
U   trunk/channels/chan_local.c

------------------------------------------------------------------------
r214945 | tilghman | 2009-08-31 11:19:16 -0500 (Mon, 31 Aug 2009) | 14 lines

Merged revisions 214940 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r214940 | tilghman | 2009-08-31 11:16:52 -0500 (Mon, 31 Aug 2009) | 7 lines
 
 Also unlock the "other" channel, when returning, due to glare.
 (closes issue ASTERISK-14727)
  Reported by: tim_ringenbach
  Patches:
        chan_local.diff uploaded by tim ringenbach (license 540)
  Tested by: tim_ringenbach
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=214945
By: Digium Subversion (svnbot) 2009-08-31 11:22:34

Repository: asterisk
Revision: 214957

_U  branches/1.6.0/
U   branches/1.6.0/channels/chan_local.c

------------------------------------------------------------------------
r214957 | tilghman | 2009-08-31 11:22:34 -0500 (Mon, 31 Aug 2009) | 21 lines

Merged revisions 214945 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r214945 | tilghman | 2009-08-31 11:18:33 -0500 (Mon, 31 Aug 2009) | 14 lines
 
 Merged revisions 214940 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r214940 | tilghman | 2009-08-31 11:16:52 -0500 (Mon, 31 Aug 2009) | 7 lines
   
   Also unlock the "other" channel, when returning, due to glare.
   (closes issue ASTERISK-14727)
    Reported by: tim_ringenbach
    Patches:
          chan_local.diff uploaded by tim ringenbach (license 540)
    Tested by: tim_ringenbach
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=214957
By: Digium Subversion (svnbot) 2009-08-31 11:22:45

Repository: asterisk
Revision: 214958

_U  branches/1.6.1/
U   branches/1.6.1/channels/chan_local.c

------------------------------------------------------------------------
r214958 | tilghman | 2009-08-31 11:22:45 -0500 (Mon, 31 Aug 2009) | 21 lines

Merged revisions 214945 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r214945 | tilghman | 2009-08-31 11:18:33 -0500 (Mon, 31 Aug 2009) | 14 lines
 
 Merged revisions 214940 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r214940 | tilghman | 2009-08-31 11:16:52 -0500 (Mon, 31 Aug 2009) | 7 lines
   
   Also unlock the "other" channel, when returning, due to glare.
   (closes issue ASTERISK-14727)
    Reported by: tim_ringenbach
    Patches:
          chan_local.diff uploaded by tim ringenbach (license 540)
    Tested by: tim_ringenbach
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=214958
By: Digium Subversion (svnbot) 2009-08-31 11:22:53

Repository: asterisk
Revision: 214960

_U  branches/1.6.2/
U   branches/1.6.2/channels/chan_local.c

------------------------------------------------------------------------
r214960 | tilghman | 2009-08-31 11:22:53 -0500 (Mon, 31 Aug 2009) | 21 lines

Merged revisions 214945 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r214945 | tilghman | 2009-08-31 11:18:33 -0500 (Mon, 31 Aug 2009) | 14 lines
 
 Merged revisions 214940 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r214940 | tilghman | 2009-08-31 11:16:52 -0500 (Mon, 31 Aug 2009) | 7 lines
   
   Also unlock the "other" channel, when returning, due to glare.
   (closes issue ASTERISK-14727)
    Reported by: tim_ringenbach
    Patches:
          chan_local.diff uploaded by tim ringenbach (license 540)
    Tested by: tim_ringenbach
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=214960