Summary:ASTERISK-20394: Asterisk crashes in ast_generic_bridge()/ast_frame_free()
Reporter:Jon Christopherson (m00ster)Labels:
Date Opened:2012-09-08 12:02:12Date Closed:2017-12-12 19:38:28.000-0600
Versions: Frequency of
Environment:CentOS 6.2 64-bit on a DELL poweredge R710 with 2x six core Xeon CPU's with 32G RAMAttachments:( 0) asterisk-backtrace.txt
( 1) asterisk-backtrace.txt
( 2) asterisk-backtrace.txt
( 3) asterisk-core-show-locks.txt
( 4) dt.txt
( 5) valgrind.txt
Description:when under load asterisk has often crashed with this type of trace. ~150 channels active, and about 90 calls being handled.

Comments:By: Jon Christopherson (m00ster) 2012-09-08 12:04:43.745-0500


By: Jon Christopherson (m00ster) 2012-09-08 12:07:44.203-0500

Asterisk often is using ~12.6G virtual ram and 100M res like this:

25124 asterisk  20   0 12.6g 100m  12m S 17.7  0.3  24:41.17 asterisk

Version details:

FreePBX Version          =
Running Asterisk Version =
Asterisk Source Version  =
Dahdi Source Version     = 2.6.1+2.6.1
Libpri Source Version    = 1.4.12    
Operating System         = CentOS release 6.2 (Final)
Kernel Version           = 2.6.32-220.23.1.el6.x86_64 - 64 Bit

By: Matt Jordan (mjordan) 2012-09-10 07:46:57.333-0500

Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks!

By: Jon Christopherson (m00ster) 2012-09-10 10:00:33.233-0500

I have attached a bit of valgrind output, i wasnt able to run it under valgrind for long due to performance issues when running under the debugger, asterisk performed too slowly to keep up with the load, I do still have it compiled with the debug options, so it it shows signs of sip deadlocks or crashes again ill attach those.

By: Jon Christopherson (m00ster) 2012-09-10 11:30:56.482-0500


I uploaded a small valgrind output, I wasn't able to run asterisk under valgrind for lng due to performance issues. It slowed asterisk down to the point it couldn't keep up with call volume.

By: Jon Christopherson (m00ster) 2012-09-10 13:51:54.570-0500

Just had asterisk stop responding and hog cpu cycles again, if it doesn't crash it will do this instead and just stop processing calls.

By: Richard Mudgett (rmudgett) 2012-09-10 17:43:50.584-0500

Strange, the core show locks file shows three threads waiting for a lock that no one seems to have locked.

By: Jon Christopherson (m00ster) 2012-09-15 17:01:29.504-0500

I am still getting crashes on, still pointing to memory corruption. I tried running with valgrind, but the performance hit made it unusable. I did attach what little output i got to this report earlier.Here is the latest core dump from today when asterisk got busy.

asterisk[1848]: segfault at 7f8f0000001a ip 00000030a1279115 sp 00007f90eac9b720 error 4 in libc-2.12.so[30a1200000+189000]

By: Jon Christopherson (m00ster) 2012-09-15 18:52:35.778-0500

Another core a couple hours later, looks diffrent to me. the timing for these cores is odd.

asterisk[5018] general protection ip:30a12478de sp:7f488043a070 error:0 in libc-2.12.so[30a1200000+189000]

By: Matt Jordan (mjordan) 2012-09-17 08:30:39.831-0500

In each of the backtraces you've uploaded, at least one thread has been in multiple nested macros.  The usage of macros is deprecated for a reason - nesting macro calls can cause Asterisk to crash in interesting ways.  While the hard limit of 7 is noted on the [wiki|https://wiki.asterisk.org/wiki/display/AST/Application_Macro], it is also possible to hit a limit much sooner than that given the appropriate conditions (such as having a stack intensive application execute).

If you replace the nested macros with GoSubs, does this problem still occur?

By: Jon Christopherson (m00ster) 2012-09-17 08:50:58.215-0500


We pretty much use the dialplan that ships with freepbx 2.9, the only changes we made to it involve placing recordings in a diffrent directory structure. I do see macros used alot on the dialplan, but not sure if i see alot of nested macros.

By: Corey Farrell (coreyfarrell) 2017-12-12 19:38:28.702-0600

Bridging has been rewritten since 1.8 and the {{ast_generic_bridge}} function no longer exists.


Per the Asterisk versions page \[1\], the maintenance (bug fix) support for the Asterisk branch you are using has ended. For continued maintenance support please move to a supported branch of Asterisk. After testing with a supported branch, if you find this problem has not been resolved, please open a new issue against the latest version of that Asterisk branch.


\[1\] https://wiki.asterisk.org/wiki/display/AST/Asterisk+Versions