Summary:	ASTERISK-11909: Deadlock during processing of IAM
Reporter:	Arnd Schmitter (arnd)	Labels:
Date Opened:	2008-04-24 06:00:32	Date Closed:	2008-04-28 12:39:25
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	Channels/chan_zap
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) deadlockfix.diff
Description:	During high call volumes, the SS7 Link goes down. The reason is, that there is a potenial Deadlock Situation in chan_zap. Below is the relevant (as I can see) part of the CLI commando core show locks. Afer some looks inside the Code i came to teh opinion, that there is a race-condition in the function ss7_start_call. In the function, a new channel gets created with the flag, to start pbx on that channel. c = zt_new(p, AST_STATE_RING, 1, SUB_REAL, law, 0); After that, the Channel gets a log of SS7 related Variables set. During these operations, the deadlock happens. So, may it help, to create the channel, without starting the pbx on it, and call ast_pbx_run(c) after setting the variables ? ****** ADDITIONAL INFORMATION ****** === Thread ID: 22539168 (ss7_linkset          started at [14206] chan_zap.c setup_zap()) === ---> Lock #0 (chan_zap.c): MUTEX 9470 ss7_linkset &p->lock 0x8e5e270 (1) === ---> Lock #1 (chan_zap.c): MUTEX 9160 ss7_start_call &linkset->lock 0x66f7824 (1) === ---> Waiting for Lock #2 (pbx.c): MUTEX 7504 pbx_builtin_setvar_helper &chan->lock_dont_use 0x8f8b564 (1) === --- ---> Locked Here: channel.c line 2318 (__ast_read)                                                                                   o ... === Thread ID: 30178208 (pbx_thread           started at [ 3781] pbx.c ast_pbx_start()) === ---> Lock #0 (channel.c): MUTEX 2318 __ast_read &chan->lock_dont_use 0x8f8b564 (1) === ---> Waiting for Lock #1 (chan_zap.c): MUTEX 5239 zt_read &p->lock 0x8e5e270 (1) === --- ---> Locked Here: chan_zap.c line 9470 (ss7_linkset)
Comments:	By: Russell Bryant (russell) 2008-04-24 10:15:29 Matt, the fix here is that in channel drivers, you always _must_ grab the channel lock before a channel pvt lock.  You probably have to do some weird stuff here to unlock the pvt, lock the channel, and then lock the pvt again. By: Matthew Fredrickson (mattf) 2008-04-25 10:13:28 Arnd, just wanted to let you know that I've seen your issue.  I've got to think about how to best fix this though without causing another deadlock. By: Matthew Fredrickson (mattf) 2008-04-25 13:31:32 I just uploaded a patch that I think will fix the problem.  Please test it and see if there are anymore issues. By: Arnd Schmitter (arnd) 2008-04-25 16:30:19 I'll test the patch after the weekend. Thanks so far. By: Arnd Schmitter (arnd) 2008-04-28 09:28:51 Matt, the patch seems to work. I wasn't able to reproduce the deadlock. Thanks for the fix. By: Digium Subversion (svnbot) 2008-04-28 11:55:29 Repository: asterisk Revision: 114776 U   trunk/channels/chan_zap.c ------------------------------------------------------------------------ r114776 | mattf | 2008-04-28 11:55:19 -0500 (Mon, 28 Apr 2008) | 1 line Fix deadlock issue in chan_zap with libss7 due to channel variables being set with the channel pvt lock being held. ASTERISK-11909 ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=114776 By: Digium Subversion (svnbot) 2008-04-28 11:59:09 Repository: asterisk Revision: 114777 _U  branches/1.6.0/ U   branches/1.6.0/channels/chan_zap.c ------------------------------------------------------------------------ r114777 | mattf | 2008-04-28 11:59:06 -0500 (Mon, 28 Apr 2008) | 9 lines Merged revisions 114776 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ........ r114776 | mattf | 2008-04-28 12:00:38 -0500 (Mon, 28 Apr 2008) | 1 line Fix deadlock issue in chan_zap with libss7 due to channel variables being set with the channel pvt lock being held. ASTERISK-11909 ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=114777 By: Matthew Fredrickson (mattf) 2008-04-28 12:39:22 Applied to trunk and 1.6.0 branch.  Have fun!