[Home]

Summary:ASTERISK-13891: Ver 1.6.0.8 Seg Fault
Reporter:Mark Hulber (hulber)Labels:
Date Opened:2009-04-03 11:37:06Date Closed:2009-06-08 13:37:03
Priority:BlockerRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) gdb.txt
( 1) malloc_debug.txt
( 2) output.txt
( 3) valgrind.txt
Description:Went from 1.6.0.6 to 1.6.0.8 and resulted in segmentation fault.  
Reverted to 1.6.0.6 and back to normal.

------------------

Linux asterisk.hulber.com 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:58:24
EST 2009 x86_64 x86_64 x86_64 GNU/Linux

Apr  3 11:49:56 asterisk kernel: asterisk[3780]: segfault at
00002ce1ac0537a8 rip 0000003e980715a8 rsp 00007fff5bf00c30 error 4
Apr  3 11:50:00 asterisk kernel: asterisk[3828]: segfault at
0000000004000000 rip 0000003e980758d9 rsp 00007fffd3138ef0 error 4
Apr  3 11:50:04 asterisk kernel: asterisk[3879]: segfault at
000000000c000000 rip 0000003e980758d9 rsp 00007fffde4cf280 error 4
Apr  3 11:50:09 asterisk kernel: asterisk[3927]: segfault at
000000001c000000 rip 0000003e980758d9 rsp 00007fff2fd65b10 error 4
Apr  3 11:50:13 asterisk kernel: asterisk[3973]: segfault at
00002ce1ac04f948 rip 0000003e980715a8 rsp 00007fff6c283fb0 error 4
Apr  3 11:50:17 asterisk kernel: asterisk[4022]: segfault at
00002ce1ac0486e8 rip 0000003e980715a8 rsp 00007fff4e1d0f00 error 4
Apr  3 11:50:21 asterisk kernel: asterisk[4069]: segfault at
00002ce1ac067e28 rip 0000003e980715a8 rsp 00007fff2f3ee120 error 4

Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed
/usr/sbin/safe_asterisk: line 117:  5322 Segmentation fault      (core
dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS}
${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed
/usr/sbin/safe_asterisk: line 117:  5372 Segmentation fault      (core
dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS}
${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed
/usr/sbin/safe_asterisk: line 117:  5419 Segmentation fault      (core
dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS}
${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed
/usr/sbin/safe_asterisk: line 117:  5467 Segmentation fault      (core
dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS}
${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed
/usr/sbin/safe_asterisk: line 117:  5514 Segmentation fault      (core
dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS}
${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
Comments:By: Leif Madsen (lmadsen) 2009-04-03 11:53:44

Was this created with DONT_OPTIMIZE?

Can you tell us how you are recreating this segfault?
By: Mark Hulber (hulber) 2009-04-03 12:03:38

Standard menuselect options.  The fault is created when Asterisk starts.  I'm not doing anything special to cause it to happen.  No calls are in progress.


By: Leif Madsen (lmadsen) 2009-04-03 14:05:01

By default DONT_OPTIMIZE is not enabled -- can you please enable it and produce a backtrace with it on? You'll need to do a 'make install' after you've enabled it. Also attach the backtrace as a text file please. Thanks!
By: Jason Parker (jparker) 2009-04-03 16:56:45

I'm going to go out on a limb and propose that this is related to ASTERISK-13889.  Can you apply revision 186517 from http://svn.digium.com/svn/asterisk/branches/1.6.0/, rebuild, and try again?
By: Mark Hulber (hulber) 2009-04-03 21:57:48

Here's my status so far.  I built revision 186517 normally although I don't get a seg fault I'm getting this repeatedly:

[Apr  3 22:49:57] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:49:57] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:49:58] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:49:59] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:49:59] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:00] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:01] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:02] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:02] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:03] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:04] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:05] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:05] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:06] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:07] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:08] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:08] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!
[Apr  3 22:50:09] NOTICE[23275]: res_musiconhold.c:593 monmp3thread: Request to schedule in the past?!?!

I'm doing some shoutcast streaming:

[regular]
;mode=files
;directory=/var/lib/asterisk/moh
;random=yes
mode=custom
dir=/var/lib/asterisk/mohmp3-empty
;application=/usr/local/bin/mpg123 -q -r 8000 -f 8192 -s --mono http://69.28.128.148:80/stream/foxnews_live
application=/usr/local/bin/mpg123 -q -r 8000 -f 8192 -s --mono http://69.28.128.148:80/stream/citadelcc_WABC-AM
;application=/usr/local/bin/mpg123 -q -r 8000 -f 8192 -s --mono http://shoutcast.dkhosting.co.uk:8014/

---

In the meantime earlier today I did some minor OS update so I'm currently at the level listed below.  I will go back to the released version and see if I'm still getting the seg fault.  

Linux asterisk.hulber.com 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24 12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
By: Mark Hulber (hulber) 2009-04-03 22:01:39

It is confirmed that I'm still getting a seg fault when I use the released version of 1.6.0.8 even though I did some minor OS upgrade.  I will collect debug again with optimization turned off.
By: Mark Hulber (hulber) 2009-04-03 22:19:12

I have uploaded a backtrace of 1.6.0.8 as released with optimization off.  It looks like it's an issue in SIP communication, as you say, possibly related to that other issue.  The musiconhold issue I'm seeing in the new branch may be a new but unrelated issue, not sure.
By: Mark Hulber (hulber) 2009-04-03 22:27:53

I'm back to 1.6.0.6 and all is running normal.
By: Tilghman Lesher (tilghman) 2009-04-04 11:16:23

Looks like memory corruption.  Please try following doc/valgrind.txt.
By: Mark Hulber (hulber) 2009-04-04 14:29:21

Uploaded valgrind output from released 1.6.0.8.  I didn't get a seg fault but did see errors / warnings.
By: Jason Parker (jparker) 2009-04-04 16:54:32

The crash you are seeing is definitely fixed by 186517.  I think any further issues should be debugged with 1.6.0.8 plus that revision.
By: Mark Hulber (hulber) 2009-04-05 09:13:16

Regarding the monmp3thread, I will test or collect something if you want.  I do use this server so I don't want to go to that release permanently since the notices are coming out twice a second.
By: Mark Hulber (hulber) 2009-04-05 09:24:46

It looks like this line (598) was changed between 1.6.0.6 and 1.6.0.8 in the loop where the message is coming out in res_musiconhold.c:

NEW:                if ((strncasecmp(class->dir, "http://", 7) && strcasecmp(class->dir, "nodir")) && AST_LIST_EMPTY(&class->membe
rs))

OLD:                 if (AST_LIST_EMPTY(&class->members))
By: Joshua C. Colp (jcolp) 2009-04-21 09:05:40

The underlying issue here has already been fixed. Please file a new issue for your res_musiconhold issue. From some preliminary investigation though it looks like your system is not waking up when expected/needed to properly read in the audio from your stream.