Summary:	ASTERISK-13984: [patch] asterisk-1.6.0.9-x86_64 segfaults when leaving a voicemail internally to another extension
Reporter:	Justin Piszcz (jpiszcz)	Labels:
Date Opened:	2009-04-19 07:59:36	Date Closed:	2009-06-08 14:40:57
Priority:	Blocker	Regression?	No
Status:	Closed/Complete	Components:	Applications/app_voicemail
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) 06052009_issue14932.patch ( 1) 20090419__bug14932.diff.txt ( 2) bug14932__backtrace.txt ( 3) gdb.txt
Description:	Taking the description from the e-mail I sent: http://www.spinics.net/lists/asterisk/msg109496.html Hello, Information: gcc -v: gcc version 4.3.3 (Debian 4.3.3-3)     os: Debian/Testing Pulled latest release from asterisk site, compiled, installed it. I have a barebones configuration: $ ls -l asterisk extensions.conf modules.conf sip.conf users.conf voicemail.conf You can see them here: http://home.comcast.net/~jpiszcz/20090418/extensions.conf http://home.comcast.net/~jpiszcz/20090418/modules.conf http://home.comcast.net/~jpiszcz/20090418/sip.conf http://home.comcast.net/~jpiszcz/20090418/users.conf http://home.comcast.net/~jpiszcz/20090418/voicemail.conf When I perform the following actions, asterisk segfaults: 1. Dial *61. 2. Enter password: XXXX 3. Enter 3 for advanced options. 4. Press 5 to leave a message, press * to return to the main menu. 5. Extension: 6000 6. Please leave your message after the tone, when done, please hangup or    press the pound key (it segfaults right after it says pound key) 7. Segmentation fault   == Using SIP RTP CoS mark 5     -- Executing [*61@line1:1] VoiceMailMain("SIP/line1-01d646b0", "6001") in new stack     -- <SIP/line1-01d646b0> Playing 'vm-password.gsm' (language 'en') DTMF begin '1' received on SIP/line1-01d646b0 DTMF begin ignored '1' on SIP/line1-01d646b0 DTMF end '1' received on SIP/line1-01d646b0, duration 190 ms DTMF end passthrough '1' on SIP/line1-01d646b0 /DTMF begin '2' received on SIP/line1-01d646b0 DTMF begin ignored '2' on SIP/line1-01d646b0 DTMF end '2' received on SIP/line1-01d646b0, duration 130 ms DTMF end passthrough '2' on SIP/line1-01d646b0 DTMF begin '3' received on SIP/line1-01d646b0 DTMF begin ignored '3' on SIP/line1-01d646b0 DTMF end '3' received on SIP/line1-01d646b0, duration 130 ms DTMF end passthrough '3' on SIP/line1-01d646b0 DTMF begin '4' received on SIP/line1-01d646b0 DTMF begin ignored '4' on SIP/line1-01d646b0 DTMF end '4' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '4' on SIP/line1-01d646b0 DTMF begin '#' received on SIP/line1-01d646b0 DTMF begin ignored '#' on SIP/line1-01d646b0 DTMF end '#' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '#' on SIP/line1-01d646b0     -- <SIP/line1-01d646b0> Playing 'vm-youhave.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-no.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-messages.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-opts.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-helpexit.gsm' (language 'en') DTMF begin '3' received on SIP/line1-01d646b0 DTMF begin ignored '3' on SIP/line1-01d646b0 DTMF end '3' received on SIP/line1-01d646b0, duration 130 ms DTMF end passthrough '3' on SIP/line1-01d646b0     -- <SIP/line1-01d646b0> Playing 'vm-leavemsg.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-starmain.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-leavemsg.gsm' (language 'en') DTMF begin '5' received on SIP/line1-01d646b0 DTMF begin ignored '5' on SIP/line1-01d646b0     -- <SIP/line1-01d646b0> Playing 'vm-starmain.gsm' (language 'en') DTMF end '5' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '5' on SIP/line1-01d646b0     -- <SIP/line1-01d646b0> Playing 'vm-extension.gsm' (language 'en') DTMF begin '6' received on SIP/line1-01d646b0 DTMF begin ignored '6' on SIP/line1-01d646b0 DTMF end '6' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '6' on SIP/line1-01d646b0 DTMF begin '0' received on SIP/line1-01d646b0 DTMF begin ignored '0' on SIP/line1-01d646b0 DTMF end '0' received on SIP/line1-01d646b0, duration 130 ms DTMF end passthrough '0' on SIP/line1-01d646b0 DTMF begin '0' received on SIP/line1-01d646b0 DTMF begin ignored '0' on SIP/line1-01d646b0 DTMF end '0' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '0' on SIP/line1-01d646b0 DTMF begin '0' received on SIP/line1-01d646b0 DTMF begin ignored '0' on SIP/line1-01d646b0 DTMF end '0' received on SIP/line1-01d646b0, duration 170 ms DTMF end passthrough '0' on SIP/line1-01d646b0     -- <SIP/line1-01d646b0> Playing 'digits/6.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'digits/0.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'digits/0.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'digits/0.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'vm-intro.gsm' (language 'en')     -- <SIP/line1-01d646b0> Playing 'beep.gsm' (language 'en') Segmentation fault (core dumped) Here is a backtrace: Core was generated by `/vapp/sbin/asterisk -vvvvvvvvvvv'. Program terminated with signal 11, Segmentation fault. [New process 23729] [New process 23717] [New process 23721] [New process 23728] [New process 23718] [New process 23723] [New process 23719] [New process 23722] [New process 23720] [New process 23726] [New process 23727] #0  0x00000000004d8bb3 in tzload (name=0x536b26 "posixrules",     sp=0x7f7f84076ba0, doextend=0) at stdtime/localtime.c:292 292                             if ((strlen(p) + strlen(name) + 1) >= sizeof fullname) (gdb) bt #0  0x00000000004d8bb3 in tzload (name=0x536b26 "posixrules",     sp=0x7f7f84076ba0, doextend=0) at stdtime/localtime.c:292 #1  0x00000000004d94bf in tzparse (name=0x7f7f8406c7a5 "", sp=0x7f7f84076ba0,     lastditch=<value optimized out>) at stdtime/localtime.c:811 #2  0x00000000004d9152 in tzload (name=<value optimized out>, sp=0x881280,     doextend=1) at stdtime/localtime.c:450 #3  0x00000000004da92d in ast_tzset (zone=0x7f7f741e5bf9 "UTC")     at stdtime/localtime.c:1029 #4  0x00000000004db98c in ast_localtime (timep=0x7f7f8407c500,     tmp=0x7f7f84076ba0, zone=0x0) at stdtime/localtime.c:1142 ASTERISK-1  0x00007f7f741d332f in get_date (s=0x7f7f84086490 "0\005\210", len=256)     at app_voicemail.c:3788 ASTERISK-2  0x00007f7f741dfb1a in leave_voicemail (chan=0x87f5f0,     ext=<value optimized out>, options=0x7f7f84090c00) at app_voicemail.c:4476 ASTERISK-3  0x00007f7f741e1ab0 in forward_message (chan=0x87f5f0, context=0x0,     vms=0x7f7f84090d40, sender=0x7f7f84096e10,     fmt=0x7f7f743ee4e0 "wav49|gsm|wav", flag=1, record_gain=0 '\0')     at app_voicemail.c:5608 ASTERISK-4  0x00007f7f741e2ece in vm_execmain (chan=0x87f5f0,     data=<value optimized out>) at app_voicemail.c:7999 ASTERISK-5  0x00000000004aedf5 in pbx_extension_helper (c=0x87f5f0,     con=<value optimized out>, context=0x87f848 "line1", exten=0x87f898 "*61",     priority=1, label=0x0, callerid=0x8794b0 "anonymous", action=E_SPAWN,     found=0x7f7f8409c03c, combined_find_spawn=1) at pbx.c:942 ASTERISK-6 0x00000000004b039a in __ast_pbx_run (c=0x87f5f0, args=0x0) at pbx.c:3614 ASTERISK-7 0x00000000004b160b in pbx_thread (data=0x536b26) at pbx.c:3974 ASTERISK-8 0x00000000004e6c0c in dummy_start (data=<value optimized out>)     at utils.c:861 ASTERISK-9 0x00007f7f8b531faa in start_thread () from /lib/libpthread.so.0 ASTERISK-10 0x00007f7f87bc62bd in clone () from /lib/libc.so.6 ASTERISK-11 0x0000000000000000 in ?? () (gdb) ****** ADDITIONAL INFORMATION ****** I see in bugs.txt I need to compile w/DONT_OPTIMIZE, did that, it segfaults after you leave the voice message this time, but still segfaults nonetheless, here is the bt and bt full:  [[[moved to an upload file]]]
Comments:	By: Justin Piszcz (jpiszcz) 2009-04-19 08:00:40 Found something very interesting! When you record a voice mail that is less than 3 seconds (what I have it set to) it does not write the file out and it does not segfault either, it is only when the voice mail is longer than 3 seconds, not sure this helps but FYI. DTMF begin '#' received on SIP/line1-01d81778 DTMF begin passthrough '#' on SIP/line1-01d81778 DTMF end '#' received on SIP/line1-01d81778, duration 190 ms DTMF end accepted with begin '#' on SIP/line1-01d81778 DTMF end passthrough '#' on SIP/line1-01d81778    -- User ended message by pressing #    -- <SIP/line1-01d81778> Playing 'auth-thankyou.ulaw' (language 'en')    -- Recording was 2 seconds long but needs to be at least 3 - abandoning    -- <SIP/line1-01d81778> Playing 'vm-opts.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'vm-helpexit.ulaw' (language 'en') DTMF begin '3' received on SIP/line1-01d81778 DTMF begin ignored '3' on SIP/line1-01d81778 DTMF end '3' received on SIP/line1-01d81778, duration 170 ms DTMF end passthrough '3' on SIP/line1-01d81778    -- <SIP/line1-01d81778> Playing 'vm-leavemsg.ulaw' (language 'en') DTMF begin '5' received on SIP/line1-01d81778 DTMF begin ignored '5' on SIP/line1-01d81778 DTMF end '5' received on SIP/line1-01d81778, duration 170 ms DTMF end passthrough '5' on SIP/line1-01d81778    -- <SIP/line1-01d81778> Playing 'vm-extension.ulaw' (language 'en') DTMF begin '6' received on SIP/line1-01d81778 DTMF begin ignored '6' on SIP/line1-01d81778 DTMF end '6' received on SIP/line1-01d81778, duration 130 ms DTMF end passthrough '6' on SIP/line1-01d81778 DTMF begin '0' received on SIP/line1-01d81778 DTMF begin ignored '0' on SIP/line1-01d81778 DTMF end '0' received on SIP/line1-01d81778, duration 170 ms DTMF end passthrough '0' on SIP/line1-01d81778 DTMF begin '0' received on SIP/line1-01d81778 DTMF begin ignored '0' on SIP/line1-01d81778 DTMF end '0' received on SIP/line1-01d81778, duration 130 ms DTMF end passthrough '0' on SIP/line1-01d81778 DTMF begin '0' received on SIP/line1-01d81778 DTMF begin ignored '0' on SIP/line1-01d81778 DTMF end '0' received on SIP/line1-01d81778, duration 130 ms DTMF end passthrough '0' on SIP/line1-01d81778    -- <SIP/line1-01d81778> Playing 'digits/6.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'vm-intro.ulaw' (language 'en')    -- <SIP/line1-01d81778> Playing 'beep.ulaw' (language 'en')    -- Recording the message    -- x=0, open writing:  /app/asterisk-1.6.0.9-x86_64/var/spool/asterisk/voice mail/default/6000/tmp/EqcJX1 format: wav49, 0x1d7f650    -- x=1, open writing:  /app/asterisk-1.6.0.9-x86_64/var/spool/asterisk/voice mail/default/6000/tmp/EqcJX1 format: gsm, 0x1d7fea0    -- x=2, open writing:  /app/asterisk-1.6.0.9-x86_64/var/spool/asterisk/voice mail/default/6000/tmp/EqcJX1 format: wav, 0x1d806f0 DTMF begin '#' received on SIP/line1-01d81778 DTMF begin passthrough '#' on SIP/line1-01d81778 DTMF end '#' received on SIP/line1-01d81778, duration 130 ms DTMF end accepted with begin '#' on SIP/line1-01d81778 DTMF end passthrough '#' on SIP/line1-01d81778    -- User ended message by pressing #    -- <SIP/line1-01d81778> Playing 'auth-thankyou.ulaw' (language 'en') Segmentation fault (core dumped) By: Justin Piszcz (jpiszcz) 2009-04-19 08:16:27 Two items: First, the Debian Testing version (1.4.x) works and does not segfault. Second, trying the latest -rc, the result is the same: FYI: in voicemail.conf: ; Needed for asterisk-1.6.1.0-rc4-x86_64 searchcontexts=yes But, it still segfaults.    -- <SIP/line1-016bb398> Playing 'digits/6.ulaw' (language 'en')    -- <SIP/line1-016bb398> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-016bb398> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-016bb398> Playing 'digits/0.ulaw' (language 'en')    -- <SIP/line1-016bb398> Playing 'vm-intro.ulaw' (language 'en')    -- <SIP/line1-016bb398> Playing 'beep.ulaw' (language 'en') Segmentation fault (core dumped) [New process 23277] [New process 23278] #0  0x00000000004e6403 in tzload (name=0x548ede "posixrules",    sp=0x7fd0c72d2a40, doextend=0) at stdtime/localtime.c:292 292                             if ((strlen(p) + strlen(name) + 1) >= sizeof fullname) (gdb) bt #0  0x00000000004e6403 in tzload (name=0x548ede "posixrules",    sp=0x7fd0c72d2a40, doextend=0) at stdtime/localtime.c:292 #1  0x00000000004e6d0f in tzparse (name=0x7fd0c72c8645 "", sp=0x7fd0c72d2a40,    lastditch=<value optimized out>) at stdtime/localtime.c:811 #2  0x00000000004e69a2 in tzload (name=<value optimized out>, sp=0x16c6270,    doextend=1) at stdtime/localtime.c:450 #3  0x00000000004e817d in ast_tzset (zone=0x7fd0d3b67688 "UTC")    at stdtime/localtime.c:1029 #4  0x00000000004e91dc in ast_localtime (timep=0x7fd0c72d83a0,    tmp=0x7fd0c72d2a40, zone=0x0) at stdtime/localtime.c:1142 ASTERISK-1  0x00007fd0d3b5375f in get_date (s=0x7fd0c72e3370 "", len=256)    at app_voicemail.c:4278 ASTERISK-2  0x00007fd0d3b608d4 in leave_voicemail (chan=0x16bfbb0,    ext=<value optimized out>, options=0x7fd0c72ebb90) at app_voicemail.c:5112 ASTERISK-3  0x00007fd0d3b62d4e in forward_message (chan=0x16bfbb0, context=0x0,    vms=0x7fd0c72ebcd0, sender=0x7fd0c72f1df0,    fmt=0x7fd0d3d70740 "wav49|gsm|wav", is_new_message=1, record_gain=0 '\0',    urgent=0) at app_voicemail.c:6363 ASTERISK-4  0x00007fd0d3b64856 in vm_execmain (chan=0x16bfbb0,    data=<value optimized out>) at app_voicemail.c:9025 ASTERISK-5  0x00000000004b9d99 in pbx_extension_helper (c=0x16bfbb0,    con=<value optimized out>, context=0x16bff68 "line1",    exten=0x16bffb8 "*61", priority=1, label=0x0,    callerid=0x16c0260 "anonymous", action=E_SPAWN, found=0x7fd0c72f703c,    combined_find_spawn=1) at pbx.c:952 ASTERISK-6 0x00000000004bb9cc in __ast_pbx_run (c=0x16bfbb0, args=0x0) at pbx.c:3643 ASTERISK-7 0x00000000004bce1b in pbx_thread (data=0x548ede) at pbx.c:4019 ASTERISK-8 0x00000000004f5b5c in dummy_start (data=<value optimized out>)    at utils.c:968 ASTERISK-9 0x00007fd0e7bd7faa in start_thread () from /lib/libpthread.so.0 ASTERISK-10 0x00007fd0e6f682bd in clone () from /lib/libc.so.6 ASTERISK-11 0x0000000000000000 in ?? () (gdb) Justin. By: Tilghman Lesher (tilghman) 2009-04-19 11:19:27 I think I've figured out what the problem is, but the patch is going to seem completely weird, because I'm making the patch in app_voicemail, rather than where it crashed, in localtime.c, and the patch has absolutely nothing to do with dates or times. From the backtrace, it appears that the stack is using 252,733 bytes of space. However, we've only allocated 245,760 bytes for thread stack.  Thus, the stack usage is exceeding available stack space.  I believe that the way to fix this is to lessen our stack usage in make_email_file(), by using heap memory instead of stack memory for our substitution routines.  Thus, that is the patch that I've uploaded. Please test and verify that it a) fixes the issue, and b) does not introduce other crashes into voicemail. By: Justin Piszcz (jpiszcz) 2009-04-19 16:18:28 Against latest SVN (pulled 2 minutes ago): $ patch -p0 < ../../patch patching file apps/app_voicemail.c Hunk #1 FAILED at 3455. Hunk #2 succeeded at 4049 (offset 584 lines). Hunk #3 succeeded at 4064 with fuzz 2 (offset 584 lines). Hunk #4 FAILED at 4094. Hunk ASTERISK-1 FAILED at 4123. Hunk ASTERISK-2 FAILED at 4214. Hunk ASTERISK-3 FAILED at 4318. 5 out of 7 hunks FAILED -- saving rejects to file apps/app_voicemail.c.rej Against 1.6.0.9: $ wget 'http://bugs.digium.com/file_download.php?file_i d=22339&type=bug' -O - | patch -p0 --2009-04-19 17:12:49--  http://bugs.digium.com/file_download.php?file_id=22339& type=bug Resolving bugs.digium.com... 76.164.171.231 Connecting to bugs.digium.com|76.164.171.231|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 7768 (7.6K) [text/plain] Saving to: `STDOUT' 100%[======================================>] 7,768       --.-K/s   in 0.1s 2009-04-19 17:12:50 (59.6 KB/s) - `-' saved [7768/7768] patching file apps/app_voicemail.c Hunk #1 succeeded at 3432 (offset -23 lines). Hunk #2 succeeded at 3442 (offset -23 lines). Hunk #3 succeeded at 3457 (offset -23 lines). Hunk #4 FAILED at 3487. Hunk ASTERISK-1 FAILED at 3516. Hunk ASTERISK-2 FAILED at 3607. Hunk ASTERISK-3 succeeded at 3672 with fuzz 1 (offset -62 lines). 3 out of 7 hunks FAILED -- saving rejects to file apps/app_voicemail.c.rej -- $ patch -p0 < ../patch patching file apps/app_voicemail.c Hunk #1 succeeded at 3432 (offset -23 lines). Hunk #2 succeeded at 3442 (offset -23 lines). Hunk #3 succeeded at 3457 (offset -23 lines). Hunk #4 FAILED at 3487. Hunk ASTERISK-1 FAILED at 3516. Hunk ASTERISK-2 FAILED at 3607. Hunk ASTERISK-3 succeeded at 3672 with fuzz 1 (offset -62 lines). 3 out of 7 hunks FAILED -- saving rejects to file apps/app_voicemail.c.rej Against 1.6.1.0-rc4: $ patch -p0 < ../patch                     patching file apps/app_voicemail.c Hunk #1 FAILED at 3455. Hunk #2 succeeded at 3887 (offset 422 lines). Hunk #3 succeeded at 3902 with fuzz 2 (offset 422 lines). Hunk #4 FAILED at 3932. Hunk ASTERISK-1 FAILED at 3961. Hunk ASTERISK-2 FAILED at 4052. Hunk ASTERISK-3 FAILED at 4156. 5 out of 7 hunks FAILED -- saving rejects to file apps/app_voicemail.c.rej $ Which version to patch against? By: Tilghman Lesher (tilghman) 2009-04-19 18:29:24 Odd, are you using 1.6.0 SVN or trunk SVN?  The patch is against 1.6.0 SVN (http://svn.digium.com/svn/asterisk/branches/1.6.0/) By: Justin Piszcz (jpiszcz) 2009-04-20 08:26:40 Hi, the patch works.. 1.6.0$ patch -p0 < ~/patch patching file apps/app_voicemail.c However, the 1.6.0 branch (SVN) does not appear to compile, it has a different issue:   [CC] eagi-test.c -> eagi-test.o   [CC] strcompat.c -> strcompat.o   [LD] eagi-test.o strcompat.o -> eagi-test   [CC] eagi-sphinx-test.c -> eagi-sphinx-test.o   [LD] eagi-sphinx-test.o -> eagi-sphinx-test   [CC] chan_agent.c -> chan_agent.o   [LD] chan_agent.o -> chan_agent.so   [CC] chan_alsa.c -> chan_alsa.o   [LD] chan_alsa.o -> chan_alsa.so   [CC] chan_h323.c -> chan_h323.o make[2]: asnparser: Command not found make[2]: *** [cisco-h225.cxx] Error 127 make[2]: *** Deleting file `cisco-h225.cxx' make[1]: *** [h323/libchanh323.a] Error 2 make: *** [channels] Error 2 Justin. By: Justin Piszcz (jpiszcz) 2009-04-20 08:27:17 $ apt-file search asnparser $ $ find . | grep asnparser # 1.6.0 src $ By: Tilghman Lesher (tilghman) 2009-04-20 10:25:10 Please enter 'make menuselect' and disable the chan_h323 driver. By: Justin Piszcz (jpiszcz) 2009-04-20 17:56:11 Done, The issue persists:    -- <SIP/line1-01f40780> Playing 'digits/6.gsm' (language 'en')    -- <SIP/line1-01f40780> Playing 'digits/0.gsm' (language 'en')    -- <SIP/line1-01f40780> Playing 'digits/0.gsm' (language 'en')    -- <SIP/line1-01f40780> Playing 'digits/0.gsm' (language 'en')    -- <SIP/line1-01f40780> Playing 'vm-intro.gsm' (language 'en')    -- <SIP/line1-01f40780> Playing 'beep.gsm' (language 'en') Segmentation fault (core dumped) (gdb) (gdb) bt #0  0x00000000004d90a3 in tzload (name=0x537186 "posixrules",    sp=0x7f0a4a27bb70, doextend=0) at stdtime/localtime.c:292 #1  0x00000000004d99af in tzparse (name=0x7f0a4a271775 "", sp=0x7f0a4a27bb70,    lastditch=<value optimized out>) at stdtime/localtime.c:811 #2  0x00000000004d9642 in tzload (name=<value optimized out>, sp=0x1f47040,    doextend=1) at stdtime/localtime.c:450 #3  0x00000000004dae1d in ast_tzset (zone=0x7f0a55147597 "UTC")    at stdtime/localtime.c:1029 #4  0x00000000004dbe7c in ast_localtime (timep=0x7f0a4a2814d0,    tmp=0x7f0a4a27bb70, zone=0x0) at stdtime/localtime.c:1142 ASTERISK-1  0x00007f0a551341df in get_date (s=0x7f0a4a28b460 "Àaô\001", len=256)    at app_voicemail.c:3855 ASTERISK-2  0x00007f0a55140dad in leave_voicemail (chan=0x1f45290,    ext=<value optimized out>, options=0x7f0a4a295bd0) at app_voicemail.c:4543 ASTERISK-3  0x00007f0a551437a8 in forward_message (chan=0x1f45290, context=0x0,    vms=0x7f0a4a295d10, sender=0x7f0a4a29bde0,    fmt=0x7f0a55350500 "wav49|gsm|wav", flag=1, record_gain=0 '\0')    at app_voicemail.c:5677 ASTERISK-4  0x00007f0a55144b6e in vm_execmain (chan=0x1f45290,    data=<value optimized out>) at app_voicemail.c:8069 ASTERISK-5  0x00000000004a5e1a in pbx_exec (c=0x1f45290, app=0x7f0a6003be40,    data=0x7f0a4a29e9f0) at pbx.c:947 ASTERISK-6 0x00000000004af6e9 in pbx_extension_helper (c=0x1f45290,    con=<value optimized out>, context=0x1f454e8 "line1",    exten=0x1f45538 "*61", priority=1, label=0x0,    callerid=0x1f458e0 "anonymous", action=E_SPAWN, found=0x7f0a4a2a104c,    combined_find_spawn=1) at pbx.c:3116 ASTERISK-7 0x00000000004b0c4a in __ast_pbx_run (c=0x1f45290, args=0x0) at pbx.c:3619 ASTERISK-8 0x00000000004b1ebb in pbx_thread (data=0x537186) at pbx.c:3979 ASTERISK-9 0x00000000004e710c in dummy_start (data=<value optimized out>)    at utils.c:861 ASTERISK-10 0x00007f0a69a69faa in start_thread () from /lib/libpthread.so.0 ASTERISK-11 0x00007f0a69f502bd in clone () from /lib/libc.so.6 ASTERISK-12 0x0000000000000000 in ?? () (gdb) (gdb) bt full #0  0x00000000004d90a3 in tzload (name=0x537186 "posixrules",    sp=0x7f0a4a27bb70, doextend=0) at stdtime/localtime.c:292        fullname = '\0' <repeats 4096 times>        p = <value optimized out>        i = <value optimized out>        fid = <value optimized out>        stored = <value optimized out>        nread = <value optimized out>        u = Cannot access memory at address 0x7f0a4a2619c0 (gdb) /app/asterisk-1.6.0-trunk-x86_64# pwd Justin. By: Tilghman Lesher (tilghman) 2009-04-20 19:41:31 I need backtraces with DONT_OPTIMIZE, as stated in doc/backtrace.txt.  Also, please upload all backtraces into the file upload area, NOT pasted into bugnotes. By: Justin Piszcz (jpiszcz) 2009-04-21 06:00:16 Sorry about that, doing so now. Bug notes (from application run): DTMF end '0' received on SIP/pstn-01721a78, duration 310 ms DTMF end passthrough '0' on SIP/pstn-01721a78    -- <SIP/pstn-01721a78> Playing 'digits/6.gsm' (language 'en')    -- <SIP/pstn-01721a78> Playing 'digits/0.gsm' (language 'en')    -- <SIP/pstn-01721a78> Playing 'digits/0.gsm' (language 'en')    -- <SIP/pstn-01721a78> Playing 'digits/0.gsm' (language 'en')    -- <SIP/pstn-01721a78> Playing 'vm-intro.gsm' (language 'en')    -- <SIP/pstn-01721a78> Playing 'beep.gsm' (language 'en')    -- Recording the message    -- x=0, open writing:  /app/asterisk-1.6.0-trunk-x86_64/var/spool/asterisk/voicemail/default/6000/tmp/T3bztI format: wav49, 0x172ffb0    -- x=1, open writing:  /app/asterisk-1.6.0-trunk-x86_64/var/spool/asterisk/voicemail/default/6000/tmp/T3bztI format: gsm, 0x1730670    -- x=2, open writing:  /app/asterisk-1.6.0-trunk-x86_64/var/spool/asterisk/voicemail/default/6000/tmp/T3bztI format: wav, 0x1730d30 DTMF begin '#' received on SIP/pstn-01721a78 DTMF begin passthrough '#' on SIP/pstn-01721a78 DTMF end '#' received on SIP/pstn-01721a78, duration 310 ms DTMF end accepted with begin '#' on SIP/pstn-01721a78 DTMF end passthrough '#' on SIP/pstn-01721a78    -- User ended message by pressing #    -- <SIP/pstn-01721a78> Playing 'auth-thankyou.gsm' (language 'en') Segmentation fault (core dumped) l1:~# By: Justin Piszcz (jpiszcz) 2009-04-21 06:01:59 File uploaded: l1:~# gdb /app/asterisk-1.6.0-trunk-x86_64/sbin/asterisk ./core (gdb) set logging on Copying output to gdb.txt. gdb.txt has been attached, which followed all the necessary steps in backtrace.txt By: carel (carel) 2009-05-12 07:37:00 I had exact same problem on Asterisk 1.6.1.0 (segfault around 2 seconds into recording voicemail message). I managed to fix this by deleting old voicemail messages for the user (var/spool/asterisk/voicemail/<context>/<ext>/INBOX/*). Tried again and it worked fine since. Not sure if the problem was due to an upgrade I did from 1.6.0.6 previously. uname -a: Linux 2.6.18-128.1.6.el5xen #1 SMP Wed Apr 1 09:53:14 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux This is a production machine so not really possible to do traces, etc. Hopefully this can help others with the same issue as a work-around. Carel By: Leif Madsen (lmadsen) 2009-06-01 14:02:54 I'd like to get an update here to try and determine if the work around that carel mentions works for anyone else? By: carel (carel) 2009-06-02 03:16:29 The workaround I had was quite fiddly. It would only work for a little while and then start crashing again. I've also tried upgrading Ram from 256MB to 512MB, but still same issue. I've not been able to replicate this on our development box - both has same version of Centos, the only difference is the underlying hardware and the VPS are slightly different - same kernel, both on XenServer. I've also tried a range of different version of 1.6 including latest SVN but still had the same problem. Also worth noting that occasionally asterisk would crash for no apparent reason - we'd have no traffic at all on the box and then suddenly it crashes. Something else that was weird is that I could not reload asterisk - there'd be about a 25% chance that SIP would stop responding. I'm not sure if it is simply a case of the sip modules failing to reload (haven't checked to be honest), but tcpdump would show sip packets arrive, but not go out. Initially I thought this had something to do with the alias IP I was using so I switched it off and stopped using reload. Not sure if it is related in any way. Eventually resorted to downgrading to 1.4 and have not had any issues since. Carel By: slic (slic) 2009-06-03 17:23:33 Carel,   I'm having all sorts of weird problems with * 1.6.1.0 (straight from download, + addons) and x86_64 (on Fedora 8): although the console never freezes, I've seen, in the last 12 hours: a) cdr_addon_mysql.c: Failed to insert into database: (1136) Column count doesn't match value count at row 1. This cleared itself after restarting Asterisk. b) problems reloading/restarting (at a certain point, "restart now" did nothing at all--I was still in the CLI, core show calls said one call was active [but in fact it wasn't], had to logout and manually kill -9 the PID of asterisk) c) SIP stops responding after approx. 45 minutes (and less than three processed calls), meaning that: c1 - No SIP messages appear to be sent/received, even with SIP debug enabled c2 - sip show peers reports all peers are registered; however, I have a SPA3102 that is logging to syslog and I see that it is actually complaining that it can't register ([0]RegFail.Retry in 30 is the entry in syslog) c3 - if I initiate a call while asterisk in "hanging", all I see is   == Using SIP RTP CoS mark 5 (although I have verbose set to 6)--nothing else, no progress in dialplan, and of course call doesn't go through c4 - "reload" command does not clear this problem, once it shows up. I have to "restart now" (and this hasn't worked every time either) c5 - On another occasion, all peers appeared as not registered (they were registered a few minutes before) except for one--even restarting the IP phones didn't make them register again (this seems another instance of problem c1) I don't think SIP dumps will be of much help--when the problem shows up, all sip messages simply stop! (not much to dump anymore...) I haven't used voicemail, I have a very very simple dialplan. All this is very strange. I was using 1.4. + addons on this machine until last monday, and didn't have any problems. I also have 1.6.0.5 (without addons) on another machine, running without any problems whatsoever since January (but this is a 32-bit machine). I know this was a bit OT, but I see a common link in the x86_64 architecture here... By: carel (carel) 2009-06-03 17:31:15 We did an upgrade tonight to: # uname -a Linux  2.6.18-128.1.10.el5xen #1 SMP Thu May 7 11:07:18 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux And the problem started appearing on Asterisk 1.4.25. Exact same issue again. Got it fixed this time by recompiling Asterisk & Asterisk-addons (my guess would be it didn't like the change in kernel). However, our Asterisk 1.2.30.2 server (both the same type of Xen VPS) happily upgraded without any issues (and we've never had stability issues on this one) slic, I must say your symptoms seem a lot like the issues I've had with 1.6.x Carel By: slic (slic) 2009-06-04 04:24:05 ...forgot this, I'm running on Linux 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 18:59:49 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Last night I had to rollback to 1.4. I decided to download and recompile the latest release (1.4.25) + addons 1.4.8, instead of reverting to 1.4.23.2. As to now,   System uptime: 10 hours, 16 minutes, 53 seconds   without any problems. I'm really wondering if this is an issue with 1.6.x, with 1.6.x + x86_64 or 1.6.x + addons. Should we open another issue or continue on this thread? By: Sean Bright (seanbright) 2009-06-05 14:20:50 I have uploaded a new patch against 1.6.0 SVN (06052009_issue14932.patch).  Please test and report your results. By: Digium Subversion (svnbot) 2009-06-08 14:24:33 Repository: asterisk Revision: 199626 U   branches/1.4/include/asterisk/utils.h ------------------------------------------------------------------------ r199626 | seanbright | 2009-06-08 14:24:33 -0500 (Mon, 08 Jun 2009) | 21 lines Increase the size of our thread stack on 64 bit processors. We were setting the stack size for each thread to 240KB regardless of architecture, which meant that in some scenarios we actually had less available stack space on 64 bit processors (pointers use 8 bytes instead of 4).  So now we calculate the stack size we reserve based on the platform's __WORDSIZE, which gives us:     32 bit -> 240KB     64 bit -> 496KB    128 bit -> 1008KB (that's right, we're ready for 128 bit processors) Patch typed by me but written by several members of #asterisk-dev, including Kevin, Tilghman, and Qwell. (closes issue ASTERISK-13984) Reported by: jpiszcz Patches:      06052009_issue14932.patch uploaded by seanbright (license 71) Tested by: seanbright ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=199626 By: Digium Subversion (svnbot) 2009-06-08 14:33:10 Repository: asterisk Revision: 199630 _U  trunk/ U   trunk/include/asterisk/utils.h ------------------------------------------------------------------------ r199630 | seanbright | 2009-06-08 14:33:09 -0500 (Mon, 08 Jun 2009) | 32 lines Merged revisions 199626,199628 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........  r199626 | seanbright | 2009-06-08 15:24:32 -0400 (Mon, 08 Jun 2009) | 21 lines    Increase the size of our thread stack on 64 bit processors.    We were setting the stack size for each thread to 240KB regardless of  architecture, which meant that in some scenarios we actually had less available  stack space on 64 bit processors (pointers use 8 bytes instead of 4).  So now we  calculate the stack size we reserve based on the platform's __WORDSIZE, which  gives us:         32 bit -> 240KB       64 bit -> 496KB      128 bit -> 1008KB (that's right, we're ready for 128 bit processors)    Patch typed by me but written by several members of #asterisk-dev, including  Kevin, Tilghman, and Qwell.    (closes issue ASTERISK-13984)  Reported by: jpiszcz  Patches:        06052009_issue14932.patch uploaded by seanbright (license 71)  Tested by: seanbright ........  r199628 | seanbright | 2009-06-08 15:28:33 -0400 (Mon, 08 Jun 2009) | 2 lines    Fix a typo in the stack size calculation just introduced. ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=199630 By: Digium Subversion (svnbot) 2009-06-08 14:39:22 Repository: asterisk Revision: 199632 _U  branches/1.6.0/ U   branches/1.6.0/include/asterisk/utils.h ------------------------------------------------------------------------ r199632 | seanbright | 2009-06-08 14:39:22 -0500 (Mon, 08 Jun 2009) | 39 lines Merged revisions 199630 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................  r199630 | seanbright | 2009-06-08 15:33:09 -0400 (Mon, 08 Jun 2009) | 32 lines    Merged revisions 199626,199628 via svnmerge from  https://origsvn.digium.com/svn/asterisk/branches/1.4    ........    r199626 | seanbright | 2009-06-08 15:24:32 -0400 (Mon, 08 Jun 2009) | 21 lines        Increase the size of our thread stack on 64 bit processors.        We were setting the stack size for each thread to 240KB regardless of    architecture, which meant that in some scenarios we actually had less available    stack space on 64 bit processors (pointers use 8 bytes instead of 4).  So now we    calculate the stack size we reserve based on the platform's __WORDSIZE, which    gives us:             32 bit -> 240KB         64 bit -> 496KB        128 bit -> 1008KB (that's right, we're ready for 128 bit processors)        Patch typed by me but written by several members of #asterisk-dev, including    Kevin, Tilghman, and Qwell.        (closes issue ASTERISK-13984)    Reported by: jpiszcz    Patches:          06052009_issue14932.patch uploaded by seanbright (license 71)    Tested by: seanbright  ........    r199628 | seanbright | 2009-06-08 15:28:33 -0400 (Mon, 08 Jun 2009) | 2 lines        Fix a typo in the stack size calculation just introduced.  ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=199632 By: Digium Subversion (svnbot) 2009-06-08 14:39:27 Repository: asterisk Revision: 199633 _U  branches/1.6.1/ U   branches/1.6.1/include/asterisk/utils.h ------------------------------------------------------------------------ r199633 | seanbright | 2009-06-08 14:39:27 -0500 (Mon, 08 Jun 2009) | 39 lines Merged revisions 199630 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................  r199630 | seanbright | 2009-06-08 15:33:09 -0400 (Mon, 08 Jun 2009) | 32 lines    Merged revisions 199626,199628 via svnmerge from  https://origsvn.digium.com/svn/asterisk/branches/1.4    ........    r199626 | seanbright | 2009-06-08 15:24:32 -0400 (Mon, 08 Jun 2009) | 21 lines        Increase the size of our thread stack on 64 bit processors.        We were setting the stack size for each thread to 240KB regardless of    architecture, which meant that in some scenarios we actually had less available    stack space on 64 bit processors (pointers use 8 bytes instead of 4).  So now we    calculate the stack size we reserve based on the platform's __WORDSIZE, which    gives us:             32 bit -> 240KB         64 bit -> 496KB        128 bit -> 1008KB (that's right, we're ready for 128 bit processors)        Patch typed by me but written by several members of #asterisk-dev, including    Kevin, Tilghman, and Qwell.        (closes issue ASTERISK-13984)    Reported by: jpiszcz    Patches:          06052009_issue14932.patch uploaded by seanbright (license 71)    Tested by: seanbright  ........    r199628 | seanbright | 2009-06-08 15:28:33 -0400 (Mon, 08 Jun 2009) | 2 lines        Fix a typo in the stack size calculation just introduced.  ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=199633 By: Digium Subversion (svnbot) 2009-06-08 14:39:31 Repository: asterisk Revision: 199634 _U  branches/1.6.2/ U   branches/1.6.2/include/asterisk/utils.h ------------------------------------------------------------------------ r199634 | seanbright | 2009-06-08 14:39:31 -0500 (Mon, 08 Jun 2009) | 39 lines Merged revisions 199630 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................  r199630 | seanbright | 2009-06-08 15:33:09 -0400 (Mon, 08 Jun 2009) | 32 lines    Merged revisions 199626,199628 via svnmerge from  https://origsvn.digium.com/svn/asterisk/branches/1.4    ........    r199626 | seanbright | 2009-06-08 15:24:32 -0400 (Mon, 08 Jun 2009) | 21 lines        Increase the size of our thread stack on 64 bit processors.        We were setting the stack size for each thread to 240KB regardless of    architecture, which meant that in some scenarios we actually had less available    stack space on 64 bit processors (pointers use 8 bytes instead of 4).  So now we    calculate the stack size we reserve based on the platform's __WORDSIZE, which    gives us:             32 bit -> 240KB         64 bit -> 496KB        128 bit -> 1008KB (that's right, we're ready for 128 bit processors)        Patch typed by me but written by several members of #asterisk-dev, including    Kevin, Tilghman, and Qwell.        (closes issue ASTERISK-13984)    Reported by: jpiszcz    Patches:          06052009_issue14932.patch uploaded by seanbright (license 71)    Tested by: seanbright  ........    r199628 | seanbright | 2009-06-08 15:28:33 -0400 (Mon, 08 Jun 2009) | 2 lines        Fix a typo in the stack size calculation just introduced.  ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=199634 By: Sean Bright (seanbright) 2009-06-08 14:40:57 I was able to duplicate the crash locally on a 64 bit processor and the submitted patch resolves the crash for me.  If it still occurs in your environment, please re-open this bug. Thanks.