Summary:ASTERISK-11909: Deadlock during processing of IAM
Reporter:Arnd Schmitter (arnd)Labels:
Date Opened:2008-04-24 06:00:32Date Closed:2008-04-28 12:39:25
Versions:Frequency of
Environment:Attachments:( 0) deadlockfix.diff
Description:During high call volumes, the SS7 Link goes down.
The reason is, that there is a potenial Deadlock Situation in chan_zap.

Below is the relevant (as I can see) part of the CLI commando core show locks.

Afer some looks inside the Code i came to teh opinion, that there is a race-condition in the function ss7_start_call.

In the function, a new channel gets created with the flag, to start pbx on that channel.
c = zt_new(p, AST_STATE_RING, 1, SUB_REAL, law, 0);

After that, the Channel gets a log of SS7 related Variables set. During these operations, the deadlock happens.

So, may it help, to create the channel, without starting the pbx on it, and call ast_pbx_run(c) after setting the variables ?


=== Thread ID: 22539168 (ss7_linkset          started at [14206] chan_zap.c setup_zap())
=== ---> Lock #0 (chan_zap.c): MUTEX 9470 ss7_linkset &p->lock 0x8e5e270 (1)
=== ---> Lock #1 (chan_zap.c): MUTEX 9160 ss7_start_call &linkset->lock 0x66f7824 (1)
=== ---> Waiting for Lock #2 (pbx.c): MUTEX 7504 pbx_builtin_setvar_helper &chan->lock_dont_use 0x8f8b564 (1)
=== --- ---> Locked Here: channel.c line 2318 (__ast_read)                                                                                  

=== Thread ID: 30178208 (pbx_thread           started at [ 3781] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 2318 __ast_read &chan->lock_dont_use 0x8f8b564 (1)
=== ---> Waiting for Lock #1 (chan_zap.c): MUTEX 5239 zt_read &p->lock 0x8e5e270 (1)
=== --- ---> Locked Here: chan_zap.c line 9470 (ss7_linkset)                                                                                
Comments:By: Russell Bryant (russell) 2008-04-24 10:15:29

Matt, the fix here is that in channel drivers, you always _must_ grab the channel lock before a channel pvt lock.  You probably have to do some weird stuff here to unlock the pvt, lock the channel, and then lock the pvt again.

By: Matthew Fredrickson (mattf) 2008-04-25 10:13:28

Arnd, just wanted to let you know that I've seen your issue.  I've got to think about how to best fix this though without causing another deadlock.

By: Matthew Fredrickson (mattf) 2008-04-25 13:31:32

I just uploaded a patch that I think will fix the problem.  Please test it and see if there are anymore issues.

By: Arnd Schmitter (arnd) 2008-04-25 16:30:19

I'll test the patch after the weekend. Thanks so far.

By: Arnd Schmitter (arnd) 2008-04-28 09:28:51

Matt, the patch seems to work. I wasn't able to reproduce the deadlock.
Thanks for the fix.

By: Digium Subversion (svnbot) 2008-04-28 11:55:29

Repository: asterisk
Revision: 114776

U   trunk/channels/chan_zap.c

r114776 | mattf | 2008-04-28 11:55:19 -0500 (Mon, 28 Apr 2008) | 1 line

Fix deadlock issue in chan_zap with libss7 due to channel variables being set with the channel pvt lock being held. ASTERISK-11909


By: Digium Subversion (svnbot) 2008-04-28 11:59:09

Repository: asterisk
Revision: 114777

_U  branches/1.6.0/
U   branches/1.6.0/channels/chan_zap.c

r114777 | mattf | 2008-04-28 11:59:06 -0500 (Mon, 28 Apr 2008) | 9 lines

Merged revisions 114776 via svnmerge from

r114776 | mattf | 2008-04-28 12:00:38 -0500 (Mon, 28 Apr 2008) | 1 line

Fix deadlock issue in chan_zap with libss7 due to channel variables being set with the channel pvt lock being held. ASTERISK-11909



By: Matthew Fredrickson (mattf) 2008-04-28 12:39:22

Applied to trunk and 1.6.0 branch.  Have fun!