Summary:ASTERISK-18730: Asterisk crashes and there is asterisk Failed to start PBX :( in logfile
Reporter:Christopher Debono (ibisgroup)Labels:
Date Opened:2011-10-18 03:39:38Date Closed:2011-11-21 14:11:25.000-0600
Versions: Frequency of
Environment:asterisk compiled from source on centos 5 running in hyper V on blade serverAttachments:
Description:For some reason asterisk crashes early morning nearly everyday stating memory problems. Fisr you see the error Failed to start PBX :(  in logfile and then further down the log I find an error Unable to launch helper thread: Cannot allocate memory.

Also when connecting via asterisk -r, it says it cannot connect. The only way to get it up is doing an amportal kill
amportal start

If needed, I have a full debug of what was happening before the crash.The same problem was happening is version 1.8.4, 1.8.6 and 1.8.7.

Server info:
Centos compiled from source
Running on hyper V
hardware IBM Blade Server

Comments:By: Paul Belanger (pabelanger) 2011-10-18 12:04:13.853-0500

Thank you for your bug report. In order to move your issue forward, we require a backtrace[1] from the core file produced after the crash. Also, be sure you have DONT_OPTIMIZE enabled in menuselect within the Compiler Flags section, then:

make install

After enabling, reproduce the crash, and then execute the backtrace[1] instructions. When complete, attach that file to this issue report.

[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

By: Tom Allen (tomallenvtg) 2011-10-22 08:44:09.683-0500

I have this issue as well. It only happens in the morning somewhere between 12 and 5AM.  Asterisk does not stop running.  While this is under investigation, can anyone recommend some automation to detect the problem and then reboot the server or do an amportal - kill and restart?  Detecting if the asterisk process is running does not trigger an event since it never fails specifically. I am using Cent OS  as well.

By: Leif Madsen (lmadsen) 2011-11-01 07:56:31.422-0500

You can still perform a 'core show locks' and a backtrace on the running process when it gets into this state.

By: Leif Madsen (lmadsen) 2011-11-01 07:56:45.603-0500

Debugging deadlocks: Please select DEBUG_THREADS and DONT_OPTIMIZE in the Compiler Flags section of menuselect. Recompile and install Asterisk (i.e. make install).  This will then give you the console command "core show locks." When the symptoms of the deadlock present themselves again, please provide output of the deadlock via:

# asterisk -rx "core show locks" | tee /tmp/core-show-locks.txt
# gdb -se "asterisk" <pid of asterisk> | tee /tmp/backtrace.txt
gdb> bt
gdb> bt full
gdb> thread apply all bt

Then attach the core-show-locks.txt and backtrace.txt files to this issue. Thanks!

By: Leif Madsen (lmadsen) 2011-11-21 14:11:19.616-0600

Suspended due to lack of activity. Please request a bug marshal in #asterisk-bugs on the IRC network irc.freenode.net to reopen the issue should you have the additional information requested.  Further information can be found at http://www.asterisk.org/developers/bug-guidelines