[Home]

Summary:ASTERISK-14374: app_queue segfault
Reporter:David Brillert (aragon)Labels:
Date Opened:2009-06-25 08:26:14Date Closed:2009-09-01 15:45:47
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Applications/app_queue
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 06252009app_queue_gdb1.txt
( 1) 06252009app_queue_gdb2.txt
( 2) 06292009app_queue_chan_local.c_line575.txt
( 3) gdb_bt_16072009.txt
( 4) gdb28072009crash1.txt
( 5) segfault_app_queue.c4086.txt
( 6) valgrindcrash07082009.txt
Description:While hammer testing 1.4.26rc4 release I have multiple crashes in app_queue
System is under very high load.
4 segfaults in 8 hours

****** ADDITIONAL INFORMATION ******

bt, bt full, thread apply all bt attached
Comments:By: David Brillert (aragon) 2009-06-29 08:21:01

4 more segfaults in app_queue channel_local yesterday.
Similar backtraces in each segfault.

New backtrace uploaded.
Asterisk version is now upgraded to 1.4 SVN r204008 (Asterisk 1.4.26rc4)


By: David Brillert (aragon) 2009-06-29 15:37:19

Based on the reproducibility of this bug/crash and its service affect. I should have opened this as blocking 1.4.26

I can lab this up quickly if a developer can upload a patch...
By: David Brillert (aragon) 2009-07-02 20:17:44

Another crash, another backtrace

This bug is killing me and I cannot put *1.4.26 into production...
By: David Brillert (aragon) 2009-07-07 12:03:16

Is any other debug info required to move forward with this bug report?
By: David Brillert (aragon) 2009-07-08 08:25:36

Crashed under valgrind
Uploaded valgrindcrash07082009.txt


By: David Brillert (aragon) 2009-07-14 12:15:38

ping

Is anything else required to get this ticket assigned?
I have provided full bt's and valgrind data...
I have test environment to quickly test patch etc...
Bug is very major if not blocking to 1.4.26 GA
By: David Brillert (aragon) 2009-07-16 08:30:44

18 segfaults overnight on a very busy system.
New file gdb_bt_16072009.txt attached with bt, bt full, thread apply all bt
By: David Brillert (aragon) 2009-07-23 14:38:19

I cannot reproduce this in SVN r206273
Please close ticket
By: David Brillert (aragon) 2009-07-24 08:10:23

Unless a developer wants to look at the debug files and determine if there is a bug that should be fixed in latest SVN...
By: David Brillert (aragon) 2009-07-28 12:32:57

don't close....
Crashed again uploaded new gdb output.
By: Digium Subversion (svnbot) 2009-08-01 06:27:39

Repository: asterisk
Revision: 209879

U   branches/1.4/main/db1-ast/mpool/mpool.c

------------------------------------------------------------------------
r209879 | russell | 2009-08-01 06:27:39 -0500 (Sat, 01 Aug 2009) | 5 lines

Resolve a valgrind warning about a read from uninitialized memory.

(issue ASTERISK-14374)
Reported by: aragon

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=209879
By: Digium Subversion (svnbot) 2009-08-01 06:29:39

Repository: asterisk
Revision: 209887

_U  trunk/
U   trunk/main/db1-ast/mpool/mpool.c

------------------------------------------------------------------------
r209887 | russell | 2009-08-01 06:29:39 -0500 (Sat, 01 Aug 2009) | 12 lines

Merged revisions 209879 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r209879 | russell | 2009-08-01 06:27:25 -0500 (Sat, 01 Aug 2009) | 5 lines
 
 Resolve a valgrind warning about a read from uninitialized memory.
 
 (issue ASTERISK-14374)
 Reported by: aragon
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=209887
By: Digium Subversion (svnbot) 2009-08-01 06:31:39

Repository: asterisk
Revision: 209896

_U  branches/1.6.0/
U   branches/1.6.0/main/db1-ast/mpool/mpool.c

------------------------------------------------------------------------
r209896 | russell | 2009-08-01 06:31:38 -0500 (Sat, 01 Aug 2009) | 19 lines

Merged revisions 209887 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r209887 | russell | 2009-08-01 06:29:25 -0500 (Sat, 01 Aug 2009) | 12 lines
 
 Merged revisions 209879 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r209879 | russell | 2009-08-01 06:27:25 -0500 (Sat, 01 Aug 2009) | 5 lines
   
   Resolve a valgrind warning about a read from uninitialized memory.
   
   (issue ASTERISK-14374)
   Reported by: aragon
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=209896
By: Digium Subversion (svnbot) 2009-08-01 06:32:35

Repository: asterisk
Revision: 209900

_U  branches/1.6.1/
U   branches/1.6.1/main/db1-ast/mpool/mpool.c

------------------------------------------------------------------------
r209900 | russell | 2009-08-01 06:32:34 -0500 (Sat, 01 Aug 2009) | 19 lines

Merged revisions 209887 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r209887 | russell | 2009-08-01 06:29:25 -0500 (Sat, 01 Aug 2009) | 12 lines
 
 Merged revisions 209879 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r209879 | russell | 2009-08-01 06:27:25 -0500 (Sat, 01 Aug 2009) | 5 lines
   
   Resolve a valgrind warning about a read from uninitialized memory.
   
   (issue ASTERISK-14374)
   Reported by: aragon
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=209900
By: Digium Subversion (svnbot) 2009-08-01 06:34:01

Repository: asterisk
Revision: 209906

_U  branches/1.6.2/
U   branches/1.6.2/main/db1-ast/mpool/mpool.c

------------------------------------------------------------------------
r209906 | russell | 2009-08-01 06:34:01 -0500 (Sat, 01 Aug 2009) | 19 lines

Merged revisions 209887 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r209887 | russell | 2009-08-01 06:29:25 -0500 (Sat, 01 Aug 2009) | 12 lines
 
 Merged revisions 209879 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r209879 | russell | 2009-08-01 06:27:25 -0500 (Sat, 01 Aug 2009) | 5 lines
   
   Resolve a valgrind warning about a read from uninitialized memory.
   
   (issue ASTERISK-14374)
   Reported by: aragon
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=209906
By: Joel Vandal (jvandal) 2009-08-01 07:47:14

aragon, I'm uploading a new RPM package of Asterisk based on revision 209955 so you will be able to made some test during your vacation :D
By: David Brillert (aragon) 2009-08-03 13:33:41

Hi jvandal

I am now using patched version in lab tests.  Also I have increased concurrent calls limit to increase likely frequency of crashes.

Russell, thanks for the patch.
Is this commit supposed to fix the reported crashes or does it just suppress a warning?
By: David Brillert (aragon) 2009-08-10 23:55:30

jvandal, the test rpm still crashes except the gdb bt output is useless and I cannot see if russell's commit fixed anything related to this bug report.
By: Leif Madsen (lmadsen) 2009-08-20 16:24:17

Setting this to status of feedback while we wait for input from aragon. Thanks!
By: David Brillert (aragon) 2009-08-20 17:13:33

Leif

This is still crashing, I presume because of parent ticket 15109 and I have uploaded a valgrind trace to 15109
https://issues.asterisk.org/file_download.php?file_id=23566&type=bug

I have no idea what Russell's commit 209995 is supposed to fix.
Does it remove a warning or does it fix a crash?
Regardless, the revision does not fix the crashes although in revision 209995 the crashes are much less frequent.


By: Russell Bryant (russell) 2009-08-25 14:51:21

I have posted a patch on issue 15109 which should address this issue.  Please give it a try!

https://issues.asterisk.org/view.php?id=15109
By: David Brillert (aragon) 2009-08-31 08:08:30

Still no crashes here. with 15109 patch.
Patch looks stable and ready for commit.
By: Digium Subversion (svnbot) 2009-09-01 15:39:25

Repository: asterisk-addons
Revision: 1023

U   branches/1.4/formats/format_mp3.c

------------------------------------------------------------------------
r1023 | russell | 2009-09-01 15:38:54 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1023
By: Digium Subversion (svnbot) 2009-09-01 15:40:33

Repository: asterisk-addons
Revision: 1024

U   branches/1.6.0/formats/format_mp3.c

------------------------------------------------------------------------
r1024 | russell | 2009-09-01 15:40:16 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1024
By: Digium Subversion (svnbot) 2009-09-01 15:42:40

Repository: asterisk-addons
Revision: 1025

U   branches/1.6.1/formats/format_mp3.c

------------------------------------------------------------------------
r1025 | russell | 2009-09-01 15:42:24 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1025
By: Digium Subversion (svnbot) 2009-09-01 15:43:29

Repository: asterisk-addons
Revision: 1026

U   branches/1.6.2/formats/format_mp3.c

------------------------------------------------------------------------
r1026 | russell | 2009-09-01 15:43:13 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1026
By: Digium Subversion (svnbot) 2009-09-01 15:45:13

Repository: asterisk
Revision: 215212

U   trunk/addons/format_mp3.c

------------------------------------------------------------------------
r215212 | russell | 2009-09-01 15:44:57 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=215212
By: Digium Subversion (svnbot) 2009-09-01 15:45:45

Repository: asterisk
Revision: 215213

_U  branches/1.6.2/

------------------------------------------------------------------------
r215213 | russell | 2009-09-01 15:45:26 -0500 (Tue, 01 Sep 2009) | 51 lines

Blocked revisions 215212 via svnmerge

........
 r215212 | russell | 2009-09-01 15:44:13 -0500 (Tue, 01 Sep 2009) | 45 lines
 
 Fix memory corruption caused by format_mp3.
 
 format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
 read().  However, it lied.  This means that other parts of the code that
 attempted to make use of the offset buffer would end up corrupting the fields
 in the ast_filestream structure.  This resulted in quite a few crashes due to
 unexpected values for fields in ast_filestream.
 
 This patch closes out quite a few bugs.  However, some of these bugs have been
 open for a while and have been an area where more than one bug has been
 discussed.  So with that said, anyone that is following one of the issues
 closed here, if you still have a problem, please open a new bug report for the
 specific problem you are still having.  If you do, please ensure that the bug
 report is based on the newest version of Asterisk, and that this patch is
 applied if format_mp3 is in use.  Thanks!
 
 (closes issue ASTERISK-14129)
 Reported by: jvandal
 Tested by: aragon, russell, zerohalo, marhbere, rgj
 
 (closes issue ASTERISK-14007)
 Reported by: aragon
 
 (closes issue ASTERISK-14141)
 Reported by: axisinternet
 
 (closes issue ASTERISK-14074)
 Reported by: maxnuv
 
 (closes issue ASTERISK-14374)
 Reported by: aragon
 
 (closes issue ASTERISK-14203)
 Reported by: amorsen
 Tested by: amorsen
 
 (closes issue ASTERISK-14718)
 Reported by: jensvb
 
 (closes issue ASTERISK-14673)
 Reported by: thom4fun
 
 (closes issue ASTERISK-14428)
 Reported by: marhbere
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=215213