[Home]

Summary:ASTERISK-14673: Asterisk Chrashes Daily
Reporter:Thomas Athineou (thom4fun)Labels:
Date Opened:2009-08-18 05:44:14Date Closed:2009-09-01 15:45:51
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Channels/chan_agent
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:We use asterisk with lot of IAX and Sip Accounts, Agents and Queues. Daily chrashes the asterisk. It seems that the reason is an agent login, because GDB / Backtrace said in every dump that an agent will login!

So I changed the asterisk versions, the hardware, and the extensions.conf to set the agentlogin to a different priority, but in every crash is the actually priority at the agentlogin.
 
Short time before the asterisk crashes, we get the following lines:
[Aug 18 11:41:57] WARNING[7356] codec_gsm.c: Invalid GSM data (1)
[Aug 18 11:41:57] WARNING[7356] translate.c: gsmtolin did not update samples 0

We also get a lot of:
[Aug 18 11:35:04] NOTICE[7837] chan_iax2.c: I should never be called!

Nothing else on notices or errors.
But we dont use GSM!

We store the CDR in a pgsql database. Before we change the database server to the local asterisk machine, asterisk crashes up to 10 times a day.
So it seems to be an timing problem??

Do you have any Idea why the asterisk crashes?
Regards
Thomas


****** ADDITIONAL INFORMATION ******

(gdb) backtrace
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7d678d0 in raise () from /lib/libc.so.6
#2  0xb7d68ff3 in abort () from /lib/libc.so.6
#3  0xb7d9cf9b in __libc_message () from /lib/libc.so.6
#4  0xb7da2961 in malloc_printerr () from /lib/libc.so.6
ASTERISK-1  0xb7da4404 in free () from /lib/libc.so.6
ASTERISK-2  0xb7d93c92 in fclose@@GLIBC_2.1 () from /lib/libc.so.6
ASTERISK-3  0x0809ec28 in filestream_destructor (arg=0xb6f30478) at file.c:346
ASTERISK-4  0x0806ecd2 in ao2_ref (user_data=0xb6f30478, delta=-1) at astobj2.c:229
ASTERISK-5  0x0809d2e0 in ast_filestream_frame_freed (fr=0xb6f304a8) at file.c:1333
ASTERISK-6 0x080a255a in ast_frame_free (frame=0xb6f304a8, cache=1) at frame.c:350
ASTERISK-7 0xb7bd9e12 in moh_files_generator (chan=0xb6f01500, data=0x820de78, len=160, samples=160) at res_musiconhold.c:295
ASTERISK-8 0x08085962 in ast_read_generator_actions (chan=0xb6f01500, f=0xb6919e20) at channel.c:1997
ASTERISK-9 0x08086009 in __ast_read (chan=0xb6f01500, dropaudio=0) at channel.c:2476
ASTERISK-10 0x080875fb in ast_safe_sleep_conditional (chan=0xb6f01500, ms=442, cond=0xb73238d0 <agent_cont_sleep>, data=0x81dc8f0) at channel.c:2510
ASTERISK-11 0xb732795a in __login_exec (chan=0xb6f01500, data=<value optimized out>, callbackmode=0) at chan_agent.c:2290
ASTERISK-12 0x080cb8d0 in pbx_extension_helper (c=0xb6f01500, con=0x0, context=0xb6f01680 "kosmomedia", exten=0xb6f016d0 "9312119", priority=2, label=0x0,
   callerid=0xb6f018f8 "312119", action=E_SPAWN) at pbx.c:536
ASTERISK-13 0x080ce2a1 in __ast_pbx_run (c=0xb6f01500) at pbx.c:2283
ASTERISK-14 0x080cf41e in pbx_thread (data=0xb6f01500) at pbx.c:2599
ASTERISK-15 0x0810001b in dummy_start (data=0xb6f02338) at utils.c:856
ASTERISK-16 0xb7ef02ab in start_thread () from /lib/libpthread.so.0
ASTERISK-17 0xb7dfeb9e in clone () from /lib/libc.so.6

(exten=0xb6f016d0 "9312119", priority=2 is an agentlogin)



(gdb) list
2687                    }
2688                    if (read(sig_alert_pipe[0], &a, sizeof(a)) != sizeof(a)) {
2689                    }
2690            }
2691
2692            return NULL;
2693    }
2694
2695    int main(int argc, char *argv[])
2696    {
Comments:By: David Brillert (aragon) 2009-08-18 08:18:07

This looks like another bug in relation to 15109
By: David Woolley (davidw) 2009-08-18 10:49:10

I'd also want to investigate if it was related to ASTERISK-14129.  (Aragon, if you put a hash before the issue number, you will get a link.)

However one point to note is that agent_login remains on the stack for the complete duration of the agent's session.  Is there some other evidence that this is actually happening just as the agent is logging in, rather than in the agent's thread?

You might want to note that it is better to run with unoptimised builds if you are producing backtraces, as optimised builds have too much detail optimised out to be of use, normally.
By: Tilghman Lesher (tilghman) 2009-08-18 11:13:37

Given where this is crashing, though, this needs valgrind (see doc/valgrind.txt).
By: Russell Bryant (russell) 2009-08-25 14:51:48

I have posted a patch on issue 15109 which should address this issue.  Please give it a try!

https://issues.asterisk.org/view.php?id=15109
By: Digium Subversion (svnbot) 2009-09-01 15:40:44

Repository: asterisk-addons
Revision: 1024

U   branches/1.6.0/formats/format_mp3.c

------------------------------------------------------------------------
r1024 | russell | 2009-09-01 15:40:16 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1024
By: Digium Subversion (svnbot) 2009-09-01 15:42:45

Repository: asterisk-addons
Revision: 1025

U   branches/1.6.1/formats/format_mp3.c

------------------------------------------------------------------------
r1025 | russell | 2009-09-01 15:42:24 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1025
By: Digium Subversion (svnbot) 2009-09-01 15:43:34

Repository: asterisk-addons
Revision: 1026

U   branches/1.6.2/formats/format_mp3.c

------------------------------------------------------------------------
r1026 | russell | 2009-09-01 15:43:13 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk-addons?view=rev&revision=1026
By: Digium Subversion (svnbot) 2009-09-01 15:45:18

Repository: asterisk
Revision: 215212

U   trunk/addons/format_mp3.c

------------------------------------------------------------------------
r215212 | russell | 2009-09-01 15:44:57 -0500 (Tue, 01 Sep 2009) | 45 lines

Fix memory corruption caused by format_mp3.

format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
read().  However, it lied.  This means that other parts of the code that
attempted to make use of the offset buffer would end up corrupting the fields
in the ast_filestream structure.  This resulted in quite a few crashes due to
unexpected values for fields in ast_filestream.

This patch closes out quite a few bugs.  However, some of these bugs have been
open for a while and have been an area where more than one bug has been
discussed.  So with that said, anyone that is following one of the issues
closed here, if you still have a problem, please open a new bug report for the
specific problem you are still having.  If you do, please ensure that the bug
report is based on the newest version of Asterisk, and that this patch is
applied if format_mp3 is in use.  Thanks!

(closes issue ASTERISK-14129)
Reported by: jvandal
Tested by: aragon, russell, zerohalo, marhbere, rgj

(closes issue ASTERISK-14007)
Reported by: aragon

(closes issue ASTERISK-14141)
Reported by: axisinternet

(closes issue ASTERISK-14074)
Reported by: maxnuv

(closes issue ASTERISK-14374)
Reported by: aragon

(closes issue ASTERISK-14203)
Reported by: amorsen
Tested by: amorsen

(closes issue ASTERISK-14718)
Reported by: jensvb

(closes issue ASTERISK-14673)
Reported by: thom4fun

(closes issue ASTERISK-14428)
Reported by: marhbere

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=215212
By: Digium Subversion (svnbot) 2009-09-01 15:45:49

Repository: asterisk
Revision: 215213

_U  branches/1.6.2/

------------------------------------------------------------------------
r215213 | russell | 2009-09-01 15:45:26 -0500 (Tue, 01 Sep 2009) | 51 lines

Blocked revisions 215212 via svnmerge

........
 r215212 | russell | 2009-09-01 15:44:13 -0500 (Tue, 01 Sep 2009) | 45 lines
 
 Fix memory corruption caused by format_mp3.
 
 format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by
 read().  However, it lied.  This means that other parts of the code that
 attempted to make use of the offset buffer would end up corrupting the fields
 in the ast_filestream structure.  This resulted in quite a few crashes due to
 unexpected values for fields in ast_filestream.
 
 This patch closes out quite a few bugs.  However, some of these bugs have been
 open for a while and have been an area where more than one bug has been
 discussed.  So with that said, anyone that is following one of the issues
 closed here, if you still have a problem, please open a new bug report for the
 specific problem you are still having.  If you do, please ensure that the bug
 report is based on the newest version of Asterisk, and that this patch is
 applied if format_mp3 is in use.  Thanks!
 
 (closes issue ASTERISK-14129)
 Reported by: jvandal
 Tested by: aragon, russell, zerohalo, marhbere, rgj
 
 (closes issue ASTERISK-14007)
 Reported by: aragon
 
 (closes issue ASTERISK-14141)
 Reported by: axisinternet
 
 (closes issue ASTERISK-14074)
 Reported by: maxnuv
 
 (closes issue ASTERISK-14374)
 Reported by: aragon
 
 (closes issue ASTERISK-14203)
 Reported by: amorsen
 Tested by: amorsen
 
 (closes issue ASTERISK-14718)
 Reported by: jensvb
 
 (closes issue ASTERISK-14673)
 Reported by: thom4fun
 
 (closes issue ASTERISK-14428)
 Reported by: marhbere
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=215213