[Home]

Summary:ASTERISK-14756: crash in local_attended_transfer, likely related to moh - 1.4.26.1
Reporter:Alan Graham (zerohalo)Labels:
Date Opened:2009-09-02 09:42:53Date Closed:2009-10-08 15:03:51
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Channels/chan_sip/Transfers
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) bt_scrubbed.txt
( 1) bt_scrubbed2.txt
Description:Reopening this as new. Attended SIP transfer from Polycom UA, no queue involved.

backtrace attached.
Comments:By: Alan Graham (zerohalo) 2009-09-02 12:26:51

forgot to add that format_mp3 is not compiled or loaded.
By: Alan Graham (zerohalo) 2009-09-08 11:37:44

getting this crash at least daily, same backtraces.
By: Alan Graham (zerohalo) 2009-09-08 11:39:48

last thing in the CLI: *** glibc detected *** double free or corruption (!prev): 0x099af8d8 ***
By: Bereterbide Marcelo (marhbere) 2009-09-08 12:02:28

Hi Zerohalo, I think this is related originally with 15109, the patch isn´t aply in 1.4.26.1; I guess you would try with the patch.. and after you should post the result.

I had have open new issue: 15845 is a similar case. where have involved attended transfers
By: Alan Graham (zerohalo) 2009-09-08 12:54:16

marhbere- We aren't using format_mp3 (not compiled or loaded), so the patch in ASTERISK-14129 isn't applicable (as I noted in ASTERISK-14129).
By: Alan Graham (zerohalo) 2009-09-09 07:43:11

If a dev would like access to a machine that's recently crashed, this can be arranged.
By: Leif Madsen (lmadsen) 2009-09-10 07:41:37

You've tried this on 1.6.0.x latest from SVN and not just a release right? None of the changes from 15109 etc... have been put into any of the latest releases yet. I'd just like to verify those changes either fix, or don't fix, this issue. Thanks!
By: Alan Graham (zerohalo) 2009-09-10 08:23:11

lmadsen - we're using 1.4 branch - I haven't seen anything committed save for the patch for format_mp3 that would attempt to address this issue. I can check out the 1.4 svn trunk and test, but as we're not using the module that the patch from 15109 is for, I'm not sure that will do much - or are you replying to ASTERISK-14784?
By: Leif Madsen (lmadsen) 2009-09-10 08:29:26

You just happened to mention you were using 1.4.26.1 - I just wanted to be 100% sure we weren't tracking down an issue that had already been fixed. If you're using the latest from the branch already (post 15109 close) then this issue can remain open.

I'd just hate to spend time debugging an issue that was already closed and fixed. I realize you're not using format_mp3, but there was some discussion that it may have fixed some other issues not specifically to format_mp3.
By: Bereterbide Marcelo (marhbere) 2009-09-10 11:36:46

zerohalo, I adviced you becouse knowed that they should ask you run last patch.
It is my opinion 15109 not solve the issue as how I tried to explained in last post in 15109. We agree mp3 wasn´t involved or isn´t only involved in this issue.

Now there is a hilo in 15609 and 15719 seeking the same crash, but I think '15109' was better for clarify. and as said in https://issues.asterisk.org/view.php?id=15609#110421, I agree with that.
By: Alan Graham (zerohalo) 2009-09-10 12:36:38

lmadsen- understood. I did not manage to save the latest bt's for the svn rev, but I'm sure I'll have a new one relatively soon.
By: Alan Graham (zerohalo) 2009-09-14 14:23:56

SVN-branch-1.4-r217156M crash BT attached.
By: Alan Graham (zerohalo) 2009-09-14 15:13:02

Here's a snap of what the CLI looked like at time of crash:

   -- Executing [s@macro-vm:2] GotoIf("SIP/XXXX1119-0848-b68b4018", "0?7:3") in new stack
   -- Goto (macro-vm,s,3)
   -- Executing [s@macro-vm:3] GosubIf("SIP/XXXX1119-0848-b68b4018", "0?strip") in new stack
   -- Executing [s@macro-vm:4] Answer("SIP/XXXX1119-0848-b68b4018", "") in new stack
 == Spawn extension (macro-out, s, 50) exited non-zero on 'SIP/XXXX0163-b6949180' in macro 'out'
 == Spawn extension (xxxxxx, XXXXXXXXXX, 51) exited non-zero on 'SIP/XXXX0163-b6949180'
   -- Executing [s@xxxxxxxxxxxx:11] BackGround("SIP/gateway1-b685d768", "xxxx/xxxxxx/xxxxxx-greeting") in new stack
   -- <SIP/gateway1-b685d768> Playing 'xxxx/xxxxxx/xxxxxx-greeting' (language 'en')
   -- Started music on hold, class 'classical', on SIP/gateway2-b70b6bc8
   -- Executing [s@macro-vm:6] VoiceMail("SIP/gateway1-b6c0b640", "XXXX@xxxxx|su") in new stack
   -- <SIP/gateway1-b6c0b640> Playing '/var/spool/asterisk/voicemail/xxxx/XXXX/unavail' (language 'en')
   -- Executing [s@macro-vm:6] VoiceMail("SIP/XXXX1119-0848-b68b4018", "XXXX@xxxx|su") in new stack
   -- <SIP/XXXX1119-0848-b68b4018> Playing '/var/spool/asterisk/voicemail/xxxx/XXXX/unavail' (language 'en')
asterisk*CLI> *** glibc detected *** double free or corruption (out): 0xb64fcd00 ***

Disconnected from Asterisk server
By: Russell Bryant (russell) 2009-09-17 21:58:27

I attached a new patch to ASTERISK-14558, give it a try and see if it helps.
By: Alan Graham (zerohalo) 2009-09-24 10:27:28

russell-

So far, so good. As I have little luck reproducing this reliably in a lab, I have put this in production for the past week with no crashes. A little more time should tell.
By: Alan Graham (zerohalo) 2009-09-24 10:27:51

fyi - this is applied to 1.4.26.1 and .2
By: Alan Graham (zerohalo) 2009-09-29 10:53:49

No crashes with v1 or 2 of the patch from ASTERISK-14558 on 1.4.26.2 yet. This looks hopeful.
By: Leif Madsen (lmadsen) 2009-09-30 10:20:13

Marking this as Ready for Review as it looks promising! :)
By: Miguel Molina (coolmig) 2009-09-30 17:14:42

Excellent news! I have one question, does this issue also affect 1.6.0 or 1.6.X series? Thanks!
By: Alan Graham (zerohalo) 2009-10-07 16:54:57

I have not seen this crash again - now running on about 10 machines for multiple weeks. Previously, I couldn't get through a couple of days. It did seem to bring ASTERISK-1543609 to the surface, but I'll post notes in that bug.
By: Digium Subversion (svnbot) 2009-10-08 14:49:19

Repository: asterisk
Revision: 222878

U   branches/1.4/include/asterisk/file.h
U   branches/1.4/include/asterisk/frame.h
U   branches/1.4/main/file.c
U   branches/1.4/main/frame.c

------------------------------------------------------------------------
r222878 | russell | 2009-10-08 14:49:10 -0500 (Thu, 08 Oct 2009) | 44 lines

Make filestream frame handling safer by isolating frames before returning them.

This patch is related to a number of issues on the bug tracker that show
crashes related to freeing frames that came from a filestream.  A number of
fixes have been made over time while trying to figure out these problems, but
there re still people seeing the crash.  (Note that some of these bug reports
include information about other problems.  I am specifically addressing
the filestream frame crash here.)

I'm still not clear on what the exact problem is.  However, what is _very_
clear is that we have seen quite a few problems over time related to unexpected
behavior when we try to use embedded frames as an optimization.  In some cases,
this optimization doesn't really provide much due to improvements made in other
areas.

In this case, the patch modifies filestream handling such that the embedded frame
will not be returned.  ast_frisolate() is used to ensure that we end up with a
completely mallocd frame.  In reality, though, we will not actually have to malloc
every time.  For filestreams, the frame will almost always be allocated and freed
in the same thread.  That means that the thread local frame cache will be used.
So, going this route doesn't hurt.

With this patch in place, some people have reported success in not seeing the
crash anymore.

(SWP-150)
(AST-208)
(ABE-1834)

(issue ASTERISK-14558)
Reported by: aragon
Patches:
     filestream_frisolate-1.4.diff2.txt uploaded by russell (license 2)
Tested by: aragon, russell

(closes issue ASTERISK-14756)
Reported by: zerohalo
Tested by: zerohalo

(closes issue ASTERISK-14784)
Reported by: marhbere

Review: https://reviewboard.asterisk.org/r/386/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=222878
By: Digium Subversion (svnbot) 2009-10-08 14:55:33

Repository: asterisk
Revision: 222880

_U  trunk/
U   trunk/include/asterisk/file.h
U   trunk/include/asterisk/frame.h
U   trunk/main/file.c
U   trunk/main/frame.c

------------------------------------------------------------------------
r222880 | russell | 2009-10-08 14:55:26 -0500 (Thu, 08 Oct 2009) | 51 lines

Merged revisions 222878 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r222878 | russell | 2009-10-08 14:45:47 -0500 (Thu, 08 Oct 2009) | 44 lines
 
 Make filestream frame handling safer by isolating frames before returning them.
 
 This patch is related to a number of issues on the bug tracker that show
 crashes related to freeing frames that came from a filestream.  A number of
 fixes have been made over time while trying to figure out these problems, but
 there re still people seeing the crash.  (Note that some of these bug reports
 include information about other problems.  I am specifically addressing
 the filestream frame crash here.)
 
 I'm still not clear on what the exact problem is.  However, what is _very_
 clear is that we have seen quite a few problems over time related to unexpected
 behavior when we try to use embedded frames as an optimization.  In some cases,
 this optimization doesn't really provide much due to improvements made in other
 areas.
 
 In this case, the patch modifies filestream handling such that the embedded frame
 will not be returned.  ast_frisolate() is used to ensure that we end up with a
 completely mallocd frame.  In reality, though, we will not actually have to malloc
 every time.  For filestreams, the frame will almost always be allocated and freed
 in the same thread.  That means that the thread local frame cache will be used.
 So, going this route doesn't hurt.
 
 With this patch in place, some people have reported success in not seeing the
 crash anymore.
 
 (SWP-150)
 (AST-208)
 (ABE-1834)
 
 (issue ASTERISK-14558)
 Reported by: aragon
 Patches:
       filestream_frisolate-1.4.diff2.txt uploaded by russell (license 2)
 Tested by: aragon, russell
 
 (closes issue ASTERISK-14756)
 Reported by: zerohalo
 Tested by: zerohalo
 
 (closes issue ASTERISK-14784)
 Reported by: marhbere
 
 Review: https://reviewboard.asterisk.org/r/386/
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=222880
By: Digium Subversion (svnbot) 2009-10-08 14:57:51

Repository: asterisk
Revision: 222881

_U  branches/1.6.0/
U   branches/1.6.0/include/asterisk/file.h
U   branches/1.6.0/include/asterisk/frame.h
U   branches/1.6.0/main/file.c
U   branches/1.6.0/main/frame.c

------------------------------------------------------------------------
r222881 | russell | 2009-10-08 14:57:44 -0500 (Thu, 08 Oct 2009) | 58 lines

Merged revisions 222880 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r222880 | russell | 2009-10-08 14:52:03 -0500 (Thu, 08 Oct 2009) | 51 lines
 
 Merged revisions 222878 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r222878 | russell | 2009-10-08 14:45:47 -0500 (Thu, 08 Oct 2009) | 44 lines
   
   Make filestream frame handling safer by isolating frames before returning them.
   
   This patch is related to a number of issues on the bug tracker that show
   crashes related to freeing frames that came from a filestream.  A number of
   fixes have been made over time while trying to figure out these problems, but
   there re still people seeing the crash.  (Note that some of these bug reports
   include information about other problems.  I am specifically addressing
   the filestream frame crash here.)
   
   I'm still not clear on what the exact problem is.  However, what is _very_
   clear is that we have seen quite a few problems over time related to unexpected
   behavior when we try to use embedded frames as an optimization.  In some cases,
   this optimization doesn't really provide much due to improvements made in other
   areas.
   
   In this case, the patch modifies filestream handling such that the embedded frame
   will not be returned.  ast_frisolate() is used to ensure that we end up with a
   completely mallocd frame.  In reality, though, we will not actually have to malloc
   every time.  For filestreams, the frame will almost always be allocated and freed
   in the same thread.  That means that the thread local frame cache will be used.
   So, going this route doesn't hurt.
   
   With this patch in place, some people have reported success in not seeing the
   crash anymore.
   
   (SWP-150)
   (AST-208)
   (ABE-1834)
   
   (issue ASTERISK-14558)
   Reported by: aragon
   Patches:
         filestream_frisolate-1.4.diff2.txt uploaded by russell (license 2)
   Tested by: aragon, russell
   
   (closes issue ASTERISK-14756)
   Reported by: zerohalo
   Tested by: zerohalo
   
   (closes issue ASTERISK-14784)
   Reported by: marhbere
   
   Review: https://reviewboard.asterisk.org/r/386/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=222881
By: Digium Subversion (svnbot) 2009-10-08 15:00:59

Repository: asterisk
Revision: 222882

_U  branches/1.6.1/
U   branches/1.6.1/include/asterisk/file.h
U   branches/1.6.1/include/asterisk/frame.h
U   branches/1.6.1/main/file.c
U   branches/1.6.1/main/frame.c

------------------------------------------------------------------------
r222882 | russell | 2009-10-08 15:00:51 -0500 (Thu, 08 Oct 2009) | 58 lines

Merged revisions 222880 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r222880 | russell | 2009-10-08 14:52:03 -0500 (Thu, 08 Oct 2009) | 51 lines
 
 Merged revisions 222878 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r222878 | russell | 2009-10-08 14:45:47 -0500 (Thu, 08 Oct 2009) | 44 lines
   
   Make filestream frame handling safer by isolating frames before returning them.
   
   This patch is related to a number of issues on the bug tracker that show
   crashes related to freeing frames that came from a filestream.  A number of
   fixes have been made over time while trying to figure out these problems, but
   there re still people seeing the crash.  (Note that some of these bug reports
   include information about other problems.  I am specifically addressing
   the filestream frame crash here.)
   
   I'm still not clear on what the exact problem is.  However, what is _very_
   clear is that we have seen quite a few problems over time related to unexpected
   behavior when we try to use embedded frames as an optimization.  In some cases,
   this optimization doesn't really provide much due to improvements made in other
   areas.
   
   In this case, the patch modifies filestream handling such that the embedded frame
   will not be returned.  ast_frisolate() is used to ensure that we end up with a
   completely mallocd frame.  In reality, though, we will not actually have to malloc
   every time.  For filestreams, the frame will almost always be allocated and freed
   in the same thread.  That means that the thread local frame cache will be used.
   So, going this route doesn't hurt.
   
   With this patch in place, some people have reported success in not seeing the
   crash anymore.
   
   (SWP-150)
   (AST-208)
   (ABE-1834)
   
   (issue ASTERISK-14558)
   Reported by: aragon
   Patches:
         filestream_frisolate-1.4.diff2.txt uploaded by russell (license 2)
   Tested by: aragon, russell
   
   (closes issue ASTERISK-14756)
   Reported by: zerohalo
   Tested by: zerohalo
   
   (closes issue ASTERISK-14784)
   Reported by: marhbere
   
   Review: https://reviewboard.asterisk.org/r/386/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=222882
By: Digium Subversion (svnbot) 2009-10-08 15:03:49

Repository: asterisk
Revision: 222883

_U  branches/1.6.2/
U   branches/1.6.2/include/asterisk/file.h
U   branches/1.6.2/include/asterisk/frame.h
U   branches/1.6.2/main/file.c
U   branches/1.6.2/main/frame.c

------------------------------------------------------------------------
r222883 | russell | 2009-10-08 15:03:42 -0500 (Thu, 08 Oct 2009) | 58 lines

Merged revisions 222880 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r222880 | russell | 2009-10-08 14:52:03 -0500 (Thu, 08 Oct 2009) | 51 lines
 
 Merged revisions 222878 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r222878 | russell | 2009-10-08 14:45:47 -0500 (Thu, 08 Oct 2009) | 44 lines
   
   Make filestream frame handling safer by isolating frames before returning them.
   
   This patch is related to a number of issues on the bug tracker that show
   crashes related to freeing frames that came from a filestream.  A number of
   fixes have been made over time while trying to figure out these problems, but
   there re still people seeing the crash.  (Note that some of these bug reports
   include information about other problems.  I am specifically addressing
   the filestream frame crash here.)
   
   I'm still not clear on what the exact problem is.  However, what is _very_
   clear is that we have seen quite a few problems over time related to unexpected
   behavior when we try to use embedded frames as an optimization.  In some cases,
   this optimization doesn't really provide much due to improvements made in other
   areas.
   
   In this case, the patch modifies filestream handling such that the embedded frame
   will not be returned.  ast_frisolate() is used to ensure that we end up with a
   completely mallocd frame.  In reality, though, we will not actually have to malloc
   every time.  For filestreams, the frame will almost always be allocated and freed
   in the same thread.  That means that the thread local frame cache will be used.
   So, going this route doesn't hurt.
   
   With this patch in place, some people have reported success in not seeing the
   crash anymore.
   
   (SWP-150)
   (AST-208)
   (ABE-1834)
   
   (issue ASTERISK-14558)
   Reported by: aragon
   Patches:
         filestream_frisolate-1.4.diff2.txt uploaded by russell (license 2)
   Tested by: aragon, russell
   
   (closes issue ASTERISK-14756)
   Reported by: zerohalo
   Tested by: zerohalo
   
   (closes issue ASTERISK-14784)
   Reported by: marhbere
   
   Review: https://reviewboard.asterisk.org/r/386/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=222883