Summary:	ASTERISK-12464: [branch] Asterisk blocked when 2 or more users leave a meetme when announce user is on
Reporter:	Andrew Pepler (pep)	Labels:
Date Opened:	2008-07-27 05:46:29	Date Closed:	2008-11-12 11:58:39.000-0600
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	Applications/app_meetme
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) coreshowlocks.txt ( 1) gdb.txt
Description:	If option I is used with meetme and 2 or more participants leave a conference simultaneously, Asterisk stops working properly.  As the number of simultaneous users leaving increases, the affect on Asterisk is increased. When the users hangup, zap channels stop accepting calls until the "user has left the conference" message has played.  When several users leave simultaneously the dialplan stops executing. console commands such as meetme and core show channels will stop working.  Sometimes the response is delayed by a few seconds and sometimes nothing is returned at all. When a larger number of users disconnect at the same time, Asterisk becomes completely unusable without a restart
Comments:	By: Mark Michelson (mmichelson) 2008-07-28 16:23:46 When I hear that there's slowdown and that certain commands stop working, my first thought is that there may be a deadlock. If you can, please recompile Asterisk with DEBUG_THREADS enabled in menuselect. Then when this happens again, get the output from the CLI command "core show locks" and attach it here to this issue. Thanks! By: Andrew Pepler (pep) 2008-08-24 10:37:39 I have attached the output of "core show locks" from when this problem occurs. I did this on 1.4.21.2 By: Mark Michelson (mmichelson) 2008-08-25 10:20:06 The "core show locks" output is showing that thread 3083000720 is holding the confs lock while thread 3082509200 is attempting to acquire it. Typically with deadlocks, the problem is that both threads hold a lock that the other is attempting to acquire, but in this case it appears that the thread holding the lock is blocking in some other way and is thus not reaching the point where the lock may be unlocked. Looking at the code, the place that seems most likely to be causing the problem is the call to ast_autoservice_stop in conf_play of app_meetme.c. To help support this, it would be helpful to see a backtrace of the thread which is currently holding the lock. The easiest way to do this would be to upload a "thread apply all bt full" output from gdb when the problem occurs. Unfortunately, this is a case where the "core show locks" output was not enough to effectively determine the cause of the deadlock, so further analysis is necessary. Thanks. By: Andrew Pepler (pep) 2008-08-31 04:04:53 I have attached the "thread apply all bt full" output By: Jeff Peeler (jpeeler) 2008-10-20 17:44:58 I've created a branch for this issue: http://svn.digium.com/svn/asterisk/team/jpeeler/bug13173 Please test and confirm this improves performance and there are no unforeseen problems. By: Tony Mountifield (softins) 2008-10-22 03:41:26 This use of a thread to play announcements is long overdue, and is an approach I submitted in a more general form in bug ASTERISK-3519, but gave up in frustration trying to get it accepted. You might like to reveiw that bug report too. By: Jeff Peeler (jpeeler) 2008-10-23 18:35:16 softins: Thanks for the pointer. Use of ast_stopstream from another thread won't work though because ast_closestream frees things, such as the filename which needs to later be deleted. I've just committed an update that uses ast_softhangup to abort ast_waitstream and it seems to be working, but there is still more work to be done. Making the announce_thread handle announcements in a more generic way I'm all for, but first need to get a solid implementation. By: Digium Subversion (svnbot) 2008-11-12 11:53:42.000-0600 Repository: asterisk Revision: 156178 U   branches/1.4/apps/app_meetme.c ------------------------------------------------------------------------ r156178 | jpeeler | 2008-11-12 11:53:42 -0600 (Wed, 12 Nov 2008) | 6 lines (closes issue ASTERISK-12464) Reported by: pep This change adds an announce_thread responsible for playing announcements to an existing conference. This allows all announcing to be immediately stopped if necessary but more importantly allows other threads that need to play something to not block. There are multiple benefits to this, but the actual bug is for solving the scenario for a channel to be unusable after hang up for the entire duration of the parting announcement. The parting announcement can be extremely long depending on what the user recorded upon joining the conference. ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=156178 By: Digium Subversion (svnbot) 2008-11-12 11:58:38.000-0600 Repository: asterisk Revision: 156178 U   branches/1.4/apps/app_meetme.c ------------------------------------------------------------------------ r156178 | jpeeler | 2008-11-12 11:53:44 -0600 (Wed, 12 Nov 2008) | 8 lines (closes issue ASTERISK-12464) Reported by: pep This change adds an announce_thread responsible for playing announcements to an existing conference. This allows all announcing to be immediately stopped if necessary but more importantly allows other threads that need to play something to not block. There are multiple benefits to this, but the actual bug is for solving the scenario for a channel to be unusable after hang up for the entire duration of the parting announcement. The parting announcement can be extremely long depending on what the user recorded upon joining the conference. Reviewed by Russell on Review Board: http://reviewboard.digium.com/r/25/ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=156178