Summary:ASTERISK-18764: Asterisk stopped accepting sip registrations, and stopped logging to full.
Reporter:Nate Slinin (nslinin)Labels:
Date Opened:2011-10-28 17:45:30Date Closed:2015-03-13 19:03:06
Versions: Frequency of
is duplicated byASTERISK-21228 Deadlock in pbx_find_extension when attempting an autoservice stop due to holding the context lock
Environment:Attachments:( 0) BT1.txt
( 1) BT2.txt
( 2) lookup.agi
Description:Today my asterisk system all of a sudden stopped accepting sip peer registrations and stopped logging activity in the log. I have DNS serv records set up so my phones connected to my backup system. After a reload Asterisk cycled logs and started logging again. Sip debug on is showing nothing even though I have a phone trying to register to the IP of the server. There is no activity on the server but it is still showing channels that are up. core show channels results in what looks like a partial list of channels that were active (it doesn't say the number of channels at the end so I know its not displaying all channels):

SIP/100043-00016ee9  s@macro-dial-one:1   Up      AppDial((Outgoing Line))      
Local/7182378888@def (None)               Up      AppDial((Outgoing Line))      
Local/7182378888@def s@macro-dialout-trun Up      Dial(SIP/Transbeam/17182378888
SIP/Transbeam-00016e 0000005@ext-queues:2 Up      (None)                        
SIP/100017-00016ef9  s@macro-dial-one:1   Up      AppDial((Outgoing Line))      
SIP/Transbeam-00016e (None)               Up      AppDial((Outgoing Line))      
SIP/Transbeam-00016e 0000005@ext-queues:2 Up      Queue(0000005,t,,,150)        
SIP/Transbeam-00016e 0000012@ext-queues:1 Up      (None)                        
SIP/Transbeam-00016e 0000012@ext-queues:1 Up      (None)                        
SIP/Transbeam-00016e 0000012@ext-queues:1 Up      (None)                        
SIP/Transbeam-00016e 0000005@ext-queues:2 Up      Queue(0000005,t,,,150)        
SIP/Transbeam-00016e 0000005@ext-queues:2 Up      Queue(0000005,t,,,150)        
Local/212@from-queue 0000005@from-queue:1 Up      AppQueue((Outgoing Line))    
Local/212@from-queue s@macro-dial-one:40  Up      Dial(SIP/100017,"",trwWM(auto-
SIP/Transbeam-00016e 0000012@ext-queues:1 Up      (None)                        
SIP/Transbeam-00016e 0000012@ext-queues:1 Up      (None)                        
SIP/Transbeam-00016e 0000005@ext-queues:2 Up      Queue(0000005,t,,,150)        
Local/373@from-queue 0000005@from-queue:1 Up      AppQueue((Outgoing Line))    
Local/373@from-queue s@macro-dial-one:40  Up      Dial(SIP/100043,"",trwWM(auto-
Local/7185072600@def (None)               Up      AppDial((Outgoing Line))      
Local/7185072600@def s@macro-dialout-trun Up      Dial(SIP/Transbeam/17185072600
SIP/100022-00016f04  7068@from-internal:1 Up      Dial(Local/7189697777/n)      
Local/7189697777@def s@macro-dialout-trun Up      Dial(SIP/Transbeam/17189697777
Local/7189697777@def (None)               Up      AppDial((Outgoing Line))      
SIP/100017-00016f01  7032@from-internal:1 Up      Dial(Local/7182378888/n)      
Local/203@from-queue 0000005@from-queue:1 Up      AppQueue((Outgoing Line))    
Local/203@from-queue s@macro-dial-one:40  Up      Dial(SIP/100044,"",trwWM(auto-
SIP/Transbeam-0000fb (None)               Up      AppDial((Outgoing Line))      
SIP/Transbeam-0000fb s@macro-dialout-trun Up      Dial(SIP/Transbeam/17186436451
SIP/100044-00016ef6  s@macro-dial-one:1   Up      AppDial((Outgoing Line))      
Local/349@from-queue 0000005@from-queue:1 Up      AppQueue((Outgoing Line))    
Local/349@from-queue s@macro-dial-one:40  Up      Dial(SIP/100022,"",trwWM(auto-

Second time I run core show channels, its "locked" and not returning anything

Attached is the BT from the running process. This is a production system with a peak call volume of approximately 30 simultaneous inbound calls, so I can't run with don't optimize flag.
Comments:By: Nate Slinin (nslinin) 2011-10-28 17:47:33.398-0500


By: Nate Slinin (nslinin) 2011-10-28 17:54:38.626-0500

Second BT after core show channels, asterisk stopped responding to CLI commands. Actually upon further investigation, it's just that shell that is not getting any response from cli commands. A new shell can launch commands and get a response, but core show channels locks that new shell too. Apparently there are 3 instances of an AGI running that seem to be stuck. I am attaching the AGI now.

By: Nate Slinin (nslinin) 2011-10-28 18:20:24.739-0500

This is a simple AGI we wrote to speeddial companies based on their four digit code using the our company's database. It may be coincidence, but there are three of them running at the time asterisk stopped working.

By: Leif Madsen (lmadsen) 2011-11-01 07:40:54.255-0500

Has this happened more than once? If so, you're probably going to want to provide 'core show locks' output as well.

By: Nate Slinin (nslinin) 2011-11-01 12:39:00.166-0500

It has happened in the past. I don't have the option for core show locks because I did not compile with DEBUG THREADS. I can recompile, but being that this is a production system with a large call volume I can't compile with anything that will slow it down or can cause it to lock up. Previously I had compiled with DON'T OPTIMIZE and DEBUG THREADS to resolve another issue and it resulted in asterisk locking up every few minutes due to the call volume. If DEBUG THREADS by itself has no effect on performance than I will recompile. Please advise.

By: Eike Maier (infy1801) 2011-12-12 02:02:48.900-0600

I had almost the same problems in the past, afer "core show channels verbose" it displays the channels, but not the count of current channels and calls an the calls processed. After displaying this my console locks complete. After comparing two same installations of asterisk, i noticed that some files are missing on the asterisk machine thats locking my console. I reinstalled asterisk an this problem does not showing up any more.