[Home]

Summary:ASTERISK-13859: [patch] segfault in ast_cdr_start() at cdr.c
Reporter:caspy (caspy)Labels:
Date Opened:2009-03-30 04:34:10Date Closed:2009-06-16 14:16:28
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:CDR/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) bt_20090330_1315.txt
( 1) bt_20090402_1111.txt
( 2) bt_20090417_1313.txt
( 3) bt_20090527_1320.txt
( 4) bt_20090527_1709.txt
( 5) cdr-next-debug1.diff
Description:Program terminated with signal 11, Segmentation fault.
#0  0x0808c705 in ast_cdr_start (cdr=0x1) at cdr.c:680
680                     if (!ast_test_flag(cdr, AST_CDR_FLAG_LOCKED)) {

(gdb) bt
#0  0x0808c705 in ast_cdr_start (cdr=0x1) at cdr.c:680
#1  0x0808df1e in ast_cdr_specialized_reset (cdr=0xb79edb48, _flags=0x0) at cdr.c:1090
#2  0x080ca680 in ast_bridge_call (chan=0xb79d0760, peer=0x8486510, config=0xb5ab990c) at features.c:2548
#3  0xb751e9cc in dial_exec_full (chan=0xb79d0760, data=0xb5abbdd8, peerflags=0xb5ab9c60, continue_exec=0x0) at app_dial.c:1913
#4  0xb751f0c8 in dial_exec (chan=0xb79d0760, data=0xb5abbdd8) at app_dial.c:1969
ASTERISK-1  0x080faa6d in pbx_exec (c=0xb79d0760, app=0xb790fa30, data=0xb5abbdd8) at pbx.c:936
ASTERISK-2  0x08101bd4 in pbx_extension_helper (c=0xb79d0760, con=0x0, context=0xb79d09a8 "macro-stdexten", exten=0xb79d09f8 "s", priority=16, label=0x0, callerid=0xb6f56608 "h?C?",
   action=E_SPAWN, found=0xb5abe548, combined_find_spawn=1) at pbx.c:3105
ASTERISK-3  0x081037e1 in ast_spawn_extension (c=0xb79d0760, context=0xb79d09a8 "macro-stdexten", exten=0xb79d09f8 "s", priority=16, callerid=0xb6f56608 "h?C?", found=0xb5abe548,
   combined_find_spawn=1) at pbx.c:3600
ASTERISK-4  0xb7076750 in _macro_exec (chan=0xb79d0760, data=0xb5ac0f78, exclusive=0) at app_macro.c:333
ASTERISK-5  0xb7077f39 in macro_exec (chan=0xb79d0760, data=0xb5ac0f78) at app_macro.c:496
ASTERISK-6 0x080faa6d in pbx_exec (c=0xb79d0760, app=0xb79634a8, data=0xb5ac0f78) at pbx.c:936
ASTERISK-7 0x08101bd4 in pbx_extension_helper (c=0xb79d0760, con=0x0, context=0xb79d09a8 "macro-stdexten", exten=0xb79d09f8 "s", priority=8, label=0x0, callerid=0xb6f56608 "h?C?",
   action=E_SPAWN, found=0xb5ac33bc, combined_find_spawn=1) at pbx.c:3105
ASTERISK-8 0x081037e1 in ast_spawn_extension (c=0xb79d0760, context=0xb79d09a8 "macro-stdexten", exten=0xb79d09f8 "s", priority=8, callerid=0xb6f56608 "h?C?", found=0xb5ac33bc,
   combined_find_spawn=1) at pbx.c:3600
ASTERISK-9 0x08103e65 in __ast_pbx_run (c=0xb79d0760, args=0x0) at pbx.c:3687
ASTERISK-10 0x0810505e in pbx_thread (data=0xb79d0760) at pbx.c:3960
ASTERISK-11 0x08153db4 in dummy_start (data=0xb7955f98) at utils.c:861
ASTERISK-12 0xb7ccf240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-13 0xb7daf49e in clone () from /lib/tls/i686/cmov/libc.so.6
Comments:By: Matthew Nicholson (mnicholson) 2009-03-31 11:34:34

Can you reproduce this?
By: caspy (caspy) 2009-03-31 12:24:21

if only wait for next segfault.
no (known) way to force it.
By: Alexey (alex001) 2009-04-01 06:57:40

#0  ast_cdr_specialized_reset (cdr=0x90, _flags=0x619432d) at cdr.c:680
680                     if (!ast_test_flag(cdr, AST_CDR_FLAG_LOCKED)) {
(gdb) bt
#0  ast_cdr_specialized_reset (cdr=0x90, _flags=0x619432d) at cdr.c:680
#1  0x000001f2c336c146 in ast_bridge_call (chan=0x1f2c5006330, peer=0x6d00cc5b8650, config=0x6d00d1d92340) at features.c:2548
#2  0x00006d00e5e5e7d2 in dial_exec_full (chan=0x1f2c5006330, data=<value optimized out>, peerflags=0x6d00d1d924f0, continue_exec=0x0) at app_dial.c:1927
#3  0x00006d00e5e60ee6 in dial_exec (chan=0x49d1ee60, data=0x619432d) at app_dial.c:1983
#4  0x000001f2c33aaebf in pbx_extension_helper (c=0x1f2c5006330, con=<value optimized out>, context=0x1f2c50065c0 "macro-stdExten", exten=0x1f2c5006610 "s", priority=1, label=0x0,
   callerid=0x1f2c4225d80 "??;??\001", action=E_SPAWN, found=0x6d00d1d978ec, combined_find_spawn=1) at pbx.c:936
ASTERISK-1  0x000001f2c33ab3e0 in ast_spawn_extension (c=0x49d1ee60, context=<value optimized out>, exten=<value optimized out>, priority=<value optimized out>,
   callerid=<value optimized out>, found=<value optimized out>, combined_find_spawn=1) at pbx.c:3600
ASTERISK-2  0x00006d00e36f790b in _macro_exec (chan=0x1f2c5006330, data=<value optimized out>, exclusive=0) at app_macro.c:333
ASTERISK-3  0x000001f2c33aaebf in pbx_extension_helper (c=0x1f2c5006330, con=<value optimized out>, context=0x1f2c50065c0 "macro-stdExten", exten=0x1f2c5006610 "s", priority=1, label=0x0,
   callerid=0x1f2c4225d80 "??;??\001", action=E_SPAWN, found=0x6d00d1d9c02c, combined_find_spawn=1) at pbx.c:936
ASTERISK-4  0x000001f2c33ac56f in __ast_pbx_run (c=0x1f2c5006330, args=0x0) at pbx.c:3600
ASTERISK-5  0x000001f2c33ad90b in pbx_thread (data=0x49d1ee60) at pbx.c:3960
ASTERISK-6 0x000001f2c33e475c in dummy_start (data=<value optimized out>) at utils.c:861
ASTERISK-7 0x00006d00f2caf017 in start_thread () from /lib/libpthread.so.0
ASTERISK-8 0x00006d00f141b19d in clone () from /lib/libc.so.6


(gdb) frame 1
#1  0x000001f2c336c146 in ast_bridge_call (chan=0x1f2c5006330, peer=0x6d00cc5b8650, config=0x6d00d1d92340) at features.c:2548
2548                            ast_cdr_specialized_reset(chan_cdr,0); /* nothing changed, reset the chan_cdr  */
(gdb) print chan_cdr
$3 = (struct ast_cdr *) 0x1f2c5003a70



My system:
uname -a
Linux *** 2.6.28-hardened ASTERISK-1 SMP PREEMPT Sun Mar 29 21:44:07 MSD 2009 x86_64 Intel(R) Xeon(R) CPU 5110 @ 1.60GHz GenuineIntel GNU/Linux
gcc -v
Reading specs from /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/specs
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-4.3.2-r3/work/gcc-4.3.2/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.3.2 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.2 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.2/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.3.2/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.2/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --disable-multilib --disable-libmudflap --disable-libssp --disable-libgomp --enable-cld --disable-libgcj --enable-languages=c,c++,treelang --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.3.2-r3 p1.6, pie-10.1.5'
Thread model: posix
gcc version 4.3.2 (Gentoo 4.3.2-r3 p1.6, pie-10.1.5)
By: Leif Madsen (lmadsen) 2009-04-01 09:31:09

alex001: can you reproduce the issue? If so, please use DONT_OPTIMIZE in the Compiler Flags section of menuselect and reinstall Asterisk (make install). Thanks!
By: caspy (caspy) 2009-04-02 02:15:31

died once again:

Core was generated by `/usr/sbin/asterisk -f -g -q'.
Program terminated with signal 11, Segmentation fault.
#0  0x0808c705 in ast_cdr_start (cdr=0xde) at cdr.c:680
680                     if (!ast_test_flag(cdr, AST_CDR_FLAG_LOCKED)) {
(gdb) bt
#0  0x0808c705 in ast_cdr_start (cdr=0xde) at cdr.c:680
#1  0x0808df1e in ast_cdr_specialized_reset (cdr=0xb5b28158, _flags=0x0) at cdr.c:1090
#2  0x080ca680 in ast_bridge_call (chan=0xb6b89690, peer=0x84645f8, config=0xb648d90c) at features.c:2548
#3  0xb771b9cc in dial_exec_full (chan=0xb6b89690, data=0xb648fdd8, peerflags=0xb648dc60, continue_exec=0x0) at app_dial.c:1913
#4  0xb771c0c8 in dial_exec (chan=0xb6b89690, data=0xb648fdd8) at app_dial.c:1969
ASTERISK-1  0x080faa6d in pbx_exec (c=0xb6b89690, app=0xb7610360, data=0xb648fdd8) at pbx.c:936
ASTERISK-2  0x08101bd4 in pbx_extension_helper (c=0xb6b89690, con=0x0, context=0xb6b898d8 "macro-stdexten", exten=0xb6b89928 "s", priority=16, label=0x0,
   callerid=0xb6b184f8 "\nAppData: SIP/1297\r\nUniqueid: 1238656184.165625\rappend_history_va", action=E_SPAWN, found=0xb6492548, combined_find_spawn=1) at pbx.c:3105
ASTERISK-3  0x081037e1 in ast_spawn_extension (c=0xb6b89690, context=0xb6b898d8 "macro-stdexten", exten=0xb6b89928 "s", priority=16,
   callerid=0xb6b184f8 "\nAppData: SIP/1297\r\nUniqueid: 1238656184.165625\rappend_history_va", found=0xb6492548, combined_find_spawn=1) at pbx.c:3600
ASTERISK-4  0xb7085750 in _macro_exec (chan=0xb6b89690, data=0xb6494f78, exclusive=0) at app_macro.c:333
ASTERISK-5  0xb7086f39 in macro_exec (chan=0xb6b89690, data=0xb6494f78) at app_macro.c:496
ASTERISK-6 0x080faa6d in pbx_exec (c=0xb6b89690, app=0xb7663010, data=0xb6494f78) at pbx.c:936
ASTERISK-7 0x08101bd4 in pbx_extension_helper (c=0xb6b89690, con=0x0, context=0xb6b898d8 "macro-stdexten", exten=0xb6b89928 "s", priority=8, label=0x0,
   callerid=0xb6b184f8 "\nAppData: SIP/1297\r\nUniqueid: 1238656184.165625\rappend_history_va", action=E_SPAWN, found=0xb64973bc, combined_find_spawn=1) at pbx.c:3105
ASTERISK-8 0x081037e1 in ast_spawn_extension (c=0xb6b89690, context=0xb6b898d8 "macro-stdexten", exten=0xb6b89928 "s", priority=8,
   callerid=0xb6b184f8 "\nAppData: SIP/1297\r\nUniqueid: 1238656184.165625\rappend_history_va", found=0xb64973bc, combined_find_spawn=1) at pbx.c:3600
ASTERISK-9 0x08103e65 in __ast_pbx_run (c=0xb6b89690, args=0x0) at pbx.c:3687
ASTERISK-10 0x0810505e in pbx_thread (data=0xb6b89690) at pbx.c:3960
ASTERISK-11 0x08153db4 in dummy_start (data=0xb6b781e0) at utils.c:861
ASTERISK-12 0xb7cdd240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-13 0xb7dbd49e in clone () from /lib/tls/i686/cmov/libc.so.6
By: Matthew Nicholson (mnicholson) 2009-04-02 09:14:28

kryptolus,

Please upload the backtrace from your crash.
By: caspy (caspy) 2009-04-02 09:58:04

hmm, why 'feedback'? what else can i provide?
By: caspy (caspy) 2009-04-17 04:20:06

next one crash:

Program terminated with signal 11, Segmentation fault.
#0  0x0808c705 in ast_cdr_start (cdr=0x3337dead) at cdr.c:680
680             if (!ast_test_flag(cdr, AST_CDR_FLAG_LOCKED)) {
(gdb) bt
#0  0x0808c705 in ast_cdr_start (cdr=0x3337dead) at cdr.c:680
#1  0x0808df1e in ast_cdr_specialized_reset (cdr=0xb61f4d68, _flags=0x0) at cdr.c:1090
#2  0x080ca680 in ast_bridge_call (chan=0xb61ed020, peer=0x878aef0, config=0xb6c6090c) at features.c:2548
#3  0xb75439cc in dial_exec_full (chan=0xb61ed020, data=0xb6c62dd8, peerflags=0xb6c60c60, continue_exec=0x0) at app_dial.c:1913
#4  0xb75440c8 in dial_exec (chan=0xb61ed020, data=0xb6c62dd8) at app_dial.c:1969
ASTERISK-1  0x080faa6d in pbx_exec (c=0xb61ed020, app=0x82242a0, data=0xb6c62dd8) at pbx.c:936
ASTERISK-2  0x08101bd4 in pbx_extension_helper (c=0xb61ed020, con=0x0, context=0xb61ed268 "macro-stdexten", exten=0xb61ed2b8 "s", priority=16, label=0x0,
   callerid=0xb5f49338 "P/\0246", action=E_SPAWN, found=0xb6c65548, combined_find_spawn=1) at pbx.c:3105
ASTERISK-3  0x081037e1 in ast_spawn_extension (c=0xb61ed020, context=0xb61ed268 "macro-stdexten", exten=0xb61ed2b8 "s", priority=16, callerid=0xb5f49338 "P/\0246", found=0xb6c65548,
   combined_find_spawn=1) at pbx.c:3600
ASTERISK-4  0xb719c750 in _macro_exec (chan=0xb61ed020, data=0xb6c67f78, exclusive=0) at app_macro.c:333
ASTERISK-5  0xb719df39 in macro_exec (chan=0xb61ed020, data=0xb6c67f78) at app_macro.c:496
ASTERISK-6 0x080faa6d in pbx_exec (c=0xb61ed020, app=0x82a3a70, data=0xb6c67f78) at pbx.c:936
ASTERISK-7 0x08101bd4 in pbx_extension_helper (c=0xb61ed020, con=0x0, context=0xb61ed268 "macro-stdexten", exten=0xb61ed2b8 "s", priority=8, label=0x0, callerid=0xb5f49338 "P/\0246",
   action=E_SPAWN, found=0xb6c6a3bc, combined_find_spawn=1) at pbx.c:3105
ASTERISK-8 0x081037e1 in ast_spawn_extension (c=0xb61ed020, context=0xb61ed268 "macro-stdexten", exten=0xb61ed2b8 "s", priority=8, callerid=0xb5f49338 "P/\0246", found=0xb6c6a3bc,
   combined_find_spawn=1) at pbx.c:3600
ASTERISK-9 0x08103e65 in __ast_pbx_run (c=0xb61ed020, args=0x0) at pbx.c:3687
ASTERISK-10 0x0810505e in pbx_thread (data=0xb61ed020) at pbx.c:3960
ASTERISK-11 0x08153db4 in dummy_start (data=0xb61ecdd8) at utils.c:861
ASTERISK-12 0xb7cf4240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-13 0xb7dd449e in clone () from /lib/tls/i686/cmov/libc.so.6
By: Matthew Nicholson (mnicholson) 2009-04-21 13:11:28

Upload a backtrace according to the instructions in doc/backtrace.txt in the asterisk source.
By: caspy (caspy) 2009-04-21 13:45:39

3 backtraces uploaded.
first one - a bug open for, and next two - according to comments.
By: Matthew Nicholson (mnicholson) 2009-04-21 13:59:19

Something causes the cdr->next pointer to be corrupt.  Either it is getting freed and not set to NULL or it is not initilized.  Or something else.  I'll look into this.
By: Amilcar S Silvestre (amilcar) 2009-05-16 09:15:40

Any progress?
By: caspy (caspy) 2009-05-27 10:43:17

today crashed twice :(
bt's attached.
By: Matthew Nicholson (mnicholson) 2009-05-27 15:49:53

Are you using forkcdr at all?
By: caspy (caspy) 2009-05-27 16:24:27

no. forkcdr() is not used at all.
By: Matthew Nicholson (mnicholson) 2009-05-27 17:35:23

I will upload a patch to help debug this shortly.
By: Matthew Nicholson (mnicholson) 2009-05-28 16:39:46

Run with the patch I uploaded and upload your debug log in the event of a crash.
By: caspy (caspy) 2009-05-29 00:03:45

patch installed.
should i see one of these messages (even normal) somewere? i do not.
By: Matthew Nicholson (mnicholson) 2009-05-29 10:44:07

These messages should appear in the debug log.  You can enable it in logger.conf.
By: Matthew Nicholson (mnicholson) 2009-05-29 10:45:25

Please upload a 'bt full' backtrace for some of those crashes.  Instructions can be found in doc/backtrace.txt in the asterisk source.
By: caspy (caspy) 2009-06-04 07:09:43

mnicholson,
will it be ok to change LOG_DEBUG to LOG_ERROR, or you need smth else from debug log?
By: Matthew Nicholson (mnicholson) 2009-06-04 11:08:35

You could change LOG_DEBUG to LOG_ERROR.  I just didn't want to fill up the console with debug messages.
By: caspy (caspy) 2009-06-04 11:56:19

thnx.
any additional load makes such bugs appear rare.
debug log, even at level 1, - more than enought.
By: caspy (caspy) 2009-06-08 03:59:37

mnicholson,
please look for patch in ASTERISK-1501192. can it be a patch for this issue too?
cause, since that patch was installed - still no crash for both this bugs.
By: Matthew Nicholson (mnicholson) 2009-06-08 10:17:41

I still don't know exactly what is causing your crash, but if that patch fixes it, that is good news.  Please keep an eye out for additional crashes to see if that patch resolves this issue.
By: caspy (caspy) 2009-06-15 02:15:49

mnicholson,

as i said earlier, patch from issue ASTERISK-1501192 make this issue seems to be resolved. Lets consider it's a fix.
By: Leif Madsen (lmadsen) 2009-06-16 14:16:27

Closed per the reporter. Thanks!