Summary:	ASTERISK-11668: Deadlock after "core show channels" or "queues show"
Reporter:	Atis Lezdins (atis)	Labels:
Date Opened:	2008-03-17 15:12:17	Date Closed:	2008-05-27 11:42:52
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	Core/Channels
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) bt.core.28649.txt ( 1) core_show_locks_2.txt ( 2) core_show_locks.txt ( 3) locks.28649.txt ( 4) thread_apply_all_bt_full.txt
Description:	Attached "core show locks" and full backtrace.
Comments:	By: Mark Michelson (mmichelson) 2008-03-17 17:27:10 I have good news and bad news. The good news: Even though your core show locks showed no apparent deadlock, I was able to figure out where the deadlock was occurring thanks to the backtrace you uploaded. I also fixed a bug in the code which helps to generate the list of locks for the "core show locks" command so hopefully future output will be more helpful. The bad news: The culprit for this deadlock is app_chanspy and the appropriate fix is not obvious. I'm going to assign this to Russell so he can figure out what to do about it. By: Atis Lezdins (atis) 2008-03-17 20:32:59 I uploaded output of "core show locks" from SVN-branch-1.4-r109226M where it's fixed. Let me know if you need full backtraces of that. By: Mark Michelson (mmichelson) 2008-03-18 14:35:02 Hmm, my fix for core show locks doesn't seem to have done anything to help the output from this scenario. I have no idea why, but for some reason, the chanspy_ds->lock is never showing up in the core show locks output. From your backtrace I was able to see that the contention is due to locking order between the channel list lock and the chanspy_ds lock. By: Digium Subversion (svnbot) 2008-03-18 17:30:27 Repository: asterisk Revision: 109763 U   branches/1.4/apps/app_chanspy.c ------------------------------------------------------------------------ r109763 | russell | 2008-03-18 17:30:27 -0500 (Tue, 18 Mar 2008) | 3 lines Fix one place where the chanspy datastore isn't removed from a channel. (issue ASTERISK-11668, reported by atis, patch by me) ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109763 By: Digium Subversion (svnbot) 2008-03-18 17:31:54 Repository: asterisk Revision: 109764 _U  trunk/ U   trunk/apps/app_chanspy.c ------------------------------------------------------------------------ r109764 | russell | 2008-03-18 17:31:53 -0500 (Tue, 18 Mar 2008) | 11 lines Merged revisions 109763 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r109763 | russell | 2008-03-18 17:34:42 -0500 (Tue, 18 Mar 2008) | 3 lines Fix one place where the chanspy datastore isn't removed from a channel. (issue ASTERISK-11668, reported by atis, patch by me) ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109764 By: Russell Bryant (russell) 2008-03-18 17:32:20 Try that .. By: Digium Subversion (svnbot) 2008-03-18 22:59:54 Repository: asterisk Revision: 109837 _U  branches/1.6.0/ U   branches/1.6.0/apps/app_chanspy.c ------------------------------------------------------------------------ r109837 | russell | 2008-03-18 22:59:53 -0500 (Tue, 18 Mar 2008) | 19 lines Merged revisions 109764 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r109764 | russell | 2008-03-18 17:36:02 -0500 (Tue, 18 Mar 2008) | 11 lines Merged revisions 109763 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r109763 | russell | 2008-03-18 17:34:42 -0500 (Tue, 18 Mar 2008) | 3 lines Fix one place where the chanspy datastore isn't removed from a channel. (issue ASTERISK-11668, reported by atis, patch by me) ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109837 By: Atis Lezdins (atis) 2008-03-19 09:01:51 Another deadlock, this time after ~ 4 hours of successful operation (previously was less) By: Mark Michelson (mmichelson) 2008-03-19 11:03:40 Just so it's documented: Using the latest backtrace and core show locks output, the contention appears to be between threads 31 and 41. The core show locks output isn't especially helpful, but the backtrace shows that the deadlock appears to be an issue of lock order between the chanspy_ds lock and the channel list lock. By: Mark Michelson (mmichelson) 2008-05-27 11:31:17 I believe that I have solved this issue with my fix to issue 12376. The behavior seems consistent between the two issues and so I will be closing this issue at the same time as 12376. If it turns out this is incorrect, please reopen this issue. By: Digium Subversion (svnbot) 2008-05-27 11:32:34 Repository: asterisk Revision: 118365 U   branches/1.4/apps/app_chanspy.c ------------------------------------------------------------------------ r118365 | mmichelson | 2008-05-27 11:32:19 -0500 (Tue, 27 May 2008) | 14 lines Add a unique id to the datastore allocated in app_chanspy since it is possible that multiple spies may be listening to the same channel. (closes issue ASTERISK-11791) Reported by: DougUDI Patches:      12376_chanspy_uid.diff uploaded by putnopvut (license 60) Tested by: destiny6628 (closes issue ASTERISK-11668) Reported by: atis ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=118365 By: Digium Subversion (svnbot) 2008-05-27 11:37:27 Repository: asterisk Revision: 118371 _U  trunk/ U   trunk/apps/app_chanspy.c ------------------------------------------------------------------------ r118371 | mmichelson | 2008-05-27 11:37:24 -0500 (Tue, 27 May 2008) | 22 lines Merged revisions 118365 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r118365 | mmichelson | 2008-05-27 11:38:38 -0500 (Tue, 27 May 2008) | 14 lines Add a unique id to the datastore allocated in app_chanspy since it is possible that multiple spies may be listening to the same channel. (closes issue ASTERISK-11791) Reported by: DougUDI Patches:      12376_chanspy_uid.diff uploaded by putnopvut (license 60) Tested by: destiny6628 (closes issue ASTERISK-11668) Reported by: atis ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=118371 By: Digium Subversion (svnbot) 2008-05-27 11:42:52 Repository: asterisk Revision: 118382 _U  branches/1.6.0/ U   branches/1.6.0/apps/app_chanspy.c ------------------------------------------------------------------------ r118382 | mmichelson | 2008-05-27 11:42:49 -0500 (Tue, 27 May 2008) | 30 lines Merged revisions 118371 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r118371 | mmichelson | 2008-05-27 11:43:36 -0500 (Tue, 27 May 2008) | 22 lines Merged revisions 118365 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r118365 | mmichelson | 2008-05-27 11:38:38 -0500 (Tue, 27 May 2008) | 14 lines Add a unique id to the datastore allocated in app_chanspy since it is possible that multiple spies may be listening to the same channel. (closes issue ASTERISK-11791) Reported by: DougUDI Patches:      12376_chanspy_uid.diff uploaded by putnopvut (license 60) Tested by: destiny6628 (closes issue ASTERISK-11668) Reported by: atis ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=118382