Summary:ASTERISK-07454: Segfault under moderate load on a 64-bit platform
Reporter:colin westlake (colinwes)Labels:
Date Opened:2006-08-03 09:31:43Date Closed:2011-06-07 14:01:07
Versions:Frequency of
Environment:Attachments:( 0) Asterisk_SVN-trunk-r38476M_gdb_trace_2nd_Aug.txt
( 1) Trunk-r39295_gdb_traces.txt
Description:In a dialer application:
Originating calls via AMI made out overZap (quad span Digium card) by way of local.
Calls sucessfully passing AMD are then sent to a queue on a remote server ( by IAX
At a sustained dial rate ~ 1cps (using < 40 chans) a crash will normally occur within 1/2 hr. At higher dial rates frequency increases.


Not definitive, but I strongly suspect that originating more calls when all zap channels are already in use precipates a crash within a couple of mins

running on Fedora 2.6.11-1.1369_FC4smp  twin proc 3 Ghz Xeon

Other dial servers connected to the same queue server and handling exactly the same call pattern (calls are round robin between servers) but running asterisk 1.2.0-beta1 on identical hardware crash very infrequently (once every 1-2 days)
Comments:By: Serge Vecher (serge-v) 2006-08-03 09:36:51

what mods have been done to the source?

By: colin westlake (colinwes) 2006-08-03 09:52:23

Nothing other than one column added to the cdrs written my app_addon.mysql, but I don't think this can be relevant as it's also been done on all the other servers and anyway not loading this module fails to cure the problem

By: Serge Vecher (serge-v) 2006-08-03 09:58:59

anything "interesting" reported on the console prior to the crash?

By: colin westlake (colinwes) 2006-08-03 10:02:06

woops - typo - I meant cdr_addon_mysql.c  !!

By: colin westlake (colinwes) 2006-08-03 10:11:31

well, under light load there is nothing to see... it's typically calling app_AMD with default parameters. (But no error messages)

When under heavy load (I suspect being asked to originate with no zap channels available)I have noticed this sort of thing in the message log:
[Jul 31 14:58:02] ERROR[30906] chan_zap.c: !! Got reject for frame 103, retransmitting frame 103 now, updating n_r!
[Jul 31 14:58:02] ERROR[30906] chan_zap.c: !! Got reject for frame 103, retransmitting frame 104 now, updating n_r!
[Jul 31 14:58:02] WARNING[13448] app_dial.c: Unable to create channel of type 'Zap' (cause 34 - Circuit/channel congestion)
[Jul 31 14:58:02] WARNING[13464] app_dial.c: Unable to create channel of type 'Zap' (cause 34 - Circuit/channel congestion)

By: Russell Bryant (russell) 2006-08-05 16:08:05

Is this code built with or without optimizations?

I would be curious to know if running with DONT_OPTIMIZE enabled in the "Compiler Flags" section of "make menuselect" would make any difference.

By: colin westlake (colinwes) 2006-08-06 02:17:11

The particular crash in the attached backtrace was on a build with DONT_OPTIMIZE enabled, but it happens just as often with optimised code. I have also tried changing the logging verbosity, but that makes no difference either.

By: Tilghman Lesher (tilghman) 2006-08-06 21:20:26

Can you install a debug-enabled version of glibc?  I'm curious to see just where _int_malloc is crashing.

Also, is this crash caused by a segfault (signal 11) or an abort (signal 6)?  The gdb output from the beginning has been removed in your file upload.

It's worth noting that we've seen several mysterious crashes inside the glibc memory routines on 64-bit only.  I'm beginning to suspect something is wrong with the glibc implementation on this platform.

By: colin westlake (colinwes) 2006-08-07 04:37:31

OK - will do. In the meantime I tried 1.2.0-beta1 (so as to be the same as the other 3 production dial servers), but compiled on 64 bit (all the others are 32) and guess what... all machines passed 65,000 calls but the 64 bit build crashed on signal 11 after 8 hours. Unfortunately the trace will not be much good as optimizition was on... I'll experiment more with debug enabled glibc as you suggest

By: wins (winson) 2006-08-09 21:33:52

I'm having a similar problem on my 64bit FC4
I'm using asterisk to dial out using zap channels. When it reaches capacity, it seems to crash with a segfault 11
gdb /usr/sbin/asterisk -c core.xxxx  
also segfaults

GNU gdb Red Hat Linux (
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `asterisk'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libncurses.so.5...done.
Loaded symbols for /usr/lib64/libncurses.so.5
Reading symbols from /usr/lib/asterisk/modules/app_cdr.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_cdr.so
Reading symbols from /usr/lib/asterisk/modules/app_macro.so...done.
Loaded symbols for /usr/lib/asterisk/modules/app_macro.so
Reading symbols from /usr/lib/asterisk/modules/format_wav_gsm.so...done.
Loaded symbols for /usr/lib/asterisk/modules/format_wav_gsm.so
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Segmentation fault

did it die in glibc ?

I tried updating glibc to 2.3.6-3  still crashes

By: colin westlake (colinwes) 2006-08-15 10:24:54

see new trace - this time on 32 bit (2.6.12-1.1378_FC3smp) SVN-trunk-r39295

This time exiting on signal 9, otherwise seems the same as before - ie falls over at very moderate loads (2 setups per sec, 60 zap chans ) other physically identical servers running 1.2.0-beta1 (2 on 32 bit 1 on 64 bit) never fall over with the exact same call loading pattern (round robin between servers)

Only other difference is that the other three are using app_machinedetect instead of app_AMD

I jaust had another crash running in console mode - this time there was some stuff of interest in the CLI
loads of: [Aug 15 17:57:50] WARNING[32635]: chan_iax2.c:7511 socket_process: Received mini frame before first full voice frame

.... followed by quite a few:
[Aug 15 17:57:50] NOTICE[32640]: chan_iax2.c:890 __schedule_action: Out of idle IAX2 threads for scheduling!
[Aug 15 17:57:51] NOTICE[32641]: chan_iax2.c:6216 socket_read: Out of idle IAX2 threads for I/O, pausing!

followed by:
Ouch ... error while writing audio data: : Broken pipe
Warning, flexibel rate not heavily tested!
Segmentation fault

By: Joshua C. Colp (jcolp) 2006-08-16 12:35:23

Would it be possible for me to gain access so I could look at the core dump and try to track down what happened?

By: colin westlake (colinwes) 2006-08-16 13:29:31

Sorry for the delay - just saw your note. I need to get home now but will arrange some access later tonight or tomorrow morning - could you email me directly with your ip (if you have fixed address) and I'll ping you back with a login c o l i n at s y n t e c co uk

By: Joshua C. Colp (jcolp) 2006-09-06 11:15:13

I'm going to suspend this for now, a lot of things have been isolated down to app_amd including this issue. I am going to look further into the code for that module to see if it is doing anything bad.