Summary:	ASTERISK-11667: Core dump during normal usage of queues
Reporter:	Mark Hamilton (yourname)	Labels:
Date Opened:	2008-03-17 15:11:12	Date Closed:	2008-03-18 12:55:29
Priority:	Critical	Regression?	No
Status:	Closed/Complete	Components:	Resources/res_features
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) bt_full.txt ( 1) bt_full_2.txt ( 2) ps_x.txt
Description:	The queues server hosts two queues, and the calls are sent to these queues via SIP. There's a fair amount of DTMF usage in here. autofill set in queues.conf. One queue transfers the first SIP transferred calls to the other queue. And this server was doing its thing when asterisk crashed. All agents complained that they got "kicked off by the system" ****** ADDITIONAL INFORMATION ****** Dell PowerEdge 1850, dual Pentium 4 Xeon 3.2Ghz CPU, 2GB Ram bt full is attached (along with thread apply) and also present at http://pastebin.ca/946534 (only btfull)
Comments:	By: Mark Hamilton (yourname) 2008-03-17 20:13:23 Another core has been dumped. Asterisk kicked all users out of the queue (might have even restarted, does it?) Core was generated by `/usr/sbin/asterisk -f -vvvg -c'. Program terminated with signal 11, Segmentation fault. #0  ast_senddigit_end (chan=0x0, digit=56 '8', duration=100) at channel.c:2480 2480            if (chan->tech->send_digit_end) The new backtraces are being uploaded. Please note: At this time, and all times, htop reports that CPU 4 is the most used one. When all agents are logged on, the usage of CPU 4 is above 95% whereas the other 3 CPUs are hardly used. The max one of those 3 CPUs goes up to is probably 2%. If I could figure out how to make Asterisk use all these 4 CPUs that would be great. By: jmls (jmls) 2008-03-18 01:36:31 the % on a single cpu should not  be that high. When running on 40+ calls, our cpu is around 20%. Is there anything else running on that machine ? By: jamesobrien (jamesobrien) 2008-03-18 05:00:55 That sounds like a problem we used to have. Of our 4 cores, one was > 95% all the time. If you looked at the processes, we had one of our mpg123 processes sitting at 24.9% (which is the equivalent of 100% of 1 of the 4 cores). From memory, it was something like having a /etc/init.d/zaptel init script (which we forgot about) but we were starting zaptel and asterisk via rc.local (or something to that effect). Anyway we had tried to start up 2 things (one failed but everything else worked fine) but one of our mpg123 processes was permanently nuts. We killed off the zaptel init script and just used our rc.local script and all was well again. By: Mark Hamilton (yourname) 2008-03-18 09:57:46 I just uploaded the output of ps x, and from what I know, sendmail is the only other application running. Which is not accessible by world because the port is firewalled. By: Mark Hamilton (yourname) 2008-03-18 11:34:13 New update: Since one of the cores was using up close to 95-100% and the rest were doing peanuts, I made to sure to go by each and every process in htop. I first killed sendmail, nothing happened. I then killed syslogd, and then instead of 98% CPU, it fell down to around 63% CPU. syslogd was logging all ALLOWED connections through the firewall. But it still doesn't make sense as to why Asterisk isn't using all the cores! With bkruse's help yesterday, I even tried setting affinity.. but couldn't do for all as there were so many asterisk threads. By: Digium Subversion (svnbot) 2008-03-18 12:54:00 Repository: asterisk Revision: 109575 U   branches/1.4/channels/chan_agent.c ------------------------------------------------------------------------ r109575 | mmichelson | 2008-03-18 12:53:59 -0500 (Tue, 18 Mar 2008) | 6 lines Make sure an agent doesn't try to send dtmf to a NULL channel closes issue ASTERISK-11667 Reported by Yourname ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109575 By: Digium Subversion (svnbot) 2008-03-18 12:55:09 Repository: asterisk Revision: 109576 _U  trunk/ U   trunk/channels/chan_agent.c ------------------------------------------------------------------------ r109576 | mmichelson | 2008-03-18 12:55:09 -0500 (Tue, 18 Mar 2008) | 14 lines Merged revisions 109575 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r109575 | mmichelson | 2008-03-18 12:58:11 -0500 (Tue, 18 Mar 2008) | 6 lines Make sure an agent doesn't try to send dtmf to a NULL channel closes issue ASTERISK-11667 Reported by Yourname ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109576 By: Digium Subversion (svnbot) 2008-03-18 12:55:29 Repository: asterisk Revision: 109577 _U  branches/1.6.0/ U   branches/1.6.0/channels/chan_agent.c ------------------------------------------------------------------------ r109577 | mmichelson | 2008-03-18 12:55:28 -0500 (Tue, 18 Mar 2008) | 22 lines Merged revisions 109576 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r109576 | mmichelson | 2008-03-18 12:59:18 -0500 (Tue, 18 Mar 2008) | 14 lines Merged revisions 109575 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r109575 | mmichelson | 2008-03-18 12:58:11 -0500 (Tue, 18 Mar 2008) | 6 lines Make sure an agent doesn't try to send dtmf to a NULL channel closes issue ASTERISK-11667 Reported by Yourname ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=109577