| Summary: | ASTERISK-14141: out of bounds crash and core dump | ||
| Reporter: | axisinternet (axisinternet) | Labels: | |
| Date Opened: | 2009-05-15 16:02:45 | Date Closed: | 2009-09-01 15:45:43 |
| Priority: | Critical | Regression? | No |
| Status: | Closed/Complete | Components: | Resources/res_musiconhold |
| Versions: | Frequency of Occurrence | ||
| Related Issues: | |||
| Environment: | Attachments: | ( 0) crash-backtrace-core.3365_2009-05-15.txt | |
| Description: | Crash and core dump - will attach backtrace from the core dump. | ||
| Comments: | By: mhardeman (mhardeman) 2009-05-15 16:13:07 Just a friendly observation (from the backtrace alone)... This looks like the same issue as 15109. By: axisinternet (axisinternet) 2009-05-15 16:25:50 Don't see the relation to 15109 - the bt shows out-of-bounds error in pbx.c which I don't note in 15109. By: Mark Michelson (mmichelson) 2009-05-15 16:52:57 axisinternet: I think that you're right in that this is not necessarily the same issue as ASTERISK-14129. Both are issues that end with the program aborting in the filestream destructor. I think, though, that this particular issue is due to poor refcounting practices present in Asterisk 1.4.22 (which have been fixed already, btw). Issue ASTERISK-14129 is due to an odd race condition that is still present in the tip of 1.4, though. By: Russell Bryant (russell) 2009-05-19 12:58:43 axisinternet, if you get a chance to test a newer version, that would be useful, as this issue may already be fixed. By: Leif Madsen (lmadsen) 2009-06-09 08:43:24 Pinging this issue again, hoping the reporter can give the latest 1.4 SVN branch a try per russell's comment. Thanks! By: axisinternet (axisinternet) 2009-06-09 08:57:17 Since this is a production server with many business customers, I cannot be using the latest SVN branch. However, I do have 1.4.24 compiled and nearly ready to deply (testing now) and hope to be able to install it by this weekend and see if the fixes in it might not resolve the problem. By: Miguel Molina (coolmig) 2009-06-10 09:47:29 Is the potential fix for this issue included on 1.4.26-rc1 or only in 1.4 SVN branch? By: Leif Madsen (lmadsen) 2009-06-10 10:18:55 Should be in 1.4.26-rc1 as it is quite recent (about a week old). By: Digium Subversion (svnbot) 2009-06-18 10:24:34 Repository: asterisk Revision: 201600 U branches/1.4/res/res_musiconhold.c ------------------------------------------------------------------------ r201600 | russell | 2009-06-18 10:24:31 -0500 (Thu, 18 Jun 2009) | 29 lines Fix memory corruption and leakage related reloads of non files mode MoH classes. For Music on Hold classes that are not files mode, meaning that we are executing an application that will feed us audio data, we use a thread to monitor the external application and read audio from it. This thread also makes use of the MoH class object. In the MoH class destructor, we used pthread_cancel() to ask the thread to exit. Unfortunately, the code did not wait to ensure that the thread actually went away. What needed to be done is a pthread_join() to ensure that the thread fully cleans up before we proceed. By adding this one line, we resolve two significant problems: 1) Since the thread was never joined, it never fully goes away. So, on every reload of non-files mode MoH, an unused thread was sticking around. 2) There was a race condition here where the application monitoring thread could still try to access the MoH class, even though the thread executing the MoH reload has already destroyed it. (issue ASTERISK-14129) Reported by: jvandal (issue ASTERISK-14141) Reported by: axisinternet (issue ASTERISK-14203) Reported by: amorsen (issue AST-208) ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=201600 By: Russell Bryant (russell) 2009-06-18 10:26:16 Please give that patch a try and let me know if it makes the crashes go away. By: Digium Subversion (svnbot) 2009-06-18 10:27:13 Repository: asterisk Revision: 201610 _U trunk/ U trunk/res/res_musiconhold.c ------------------------------------------------------------------------ r201610 | russell | 2009-06-18 10:27:10 -0500 (Thu, 18 Jun 2009) | 36 lines Merged revisions 201600 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r201600 | russell | 2009-06-18 10:24:31 -0500 (Thu, 18 Jun 2009) | 29 lines Fix memory corruption and leakage related reloads of non files mode MoH classes. For Music on Hold classes that are not files mode, meaning that we are executing an application that will feed us audio data, we use a thread to monitor the external application and read audio from it. This thread also makes use of the MoH class object. In the MoH class destructor, we used pthread_cancel() to ask the thread to exit. Unfortunately, the code did not wait to ensure that the thread actually went away. What needed to be done is a pthread_join() to ensure that the thread fully cleans up before we proceed. By adding this one line, we resolve two significant problems: 1) Since the thread was never joined, it never fully goes away. So, on every reload of non-files mode MoH, an unused thread was sticking around. 2) There was a race condition here where the application monitoring thread could still try to access the MoH class, even though the thread executing the MoH reload has already destroyed it. (issue ASTERISK-14129) Reported by: jvandal (issue ASTERISK-14141) Reported by: axisinternet (issue ASTERISK-14203) Reported by: amorsen (issue AST-208) ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=201610 By: Digium Subversion (svnbot) 2009-06-18 10:32:41 Repository: asterisk Revision: 201612 _U branches/1.6.0/ U branches/1.6.0/res/res_musiconhold.c ------------------------------------------------------------------------ r201612 | russell | 2009-06-18 10:32:38 -0500 (Thu, 18 Jun 2009) | 43 lines Merged revisions 201610 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r201610 | russell | 2009-06-18 10:27:10 -0500 (Thu, 18 Jun 2009) | 36 lines Merged revisions 201600 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r201600 | russell | 2009-06-18 10:24:31 -0500 (Thu, 18 Jun 2009) | 29 lines Fix memory corruption and leakage related reloads of non files mode MoH classes. For Music on Hold classes that are not files mode, meaning that we are executing an application that will feed us audio data, we use a thread to monitor the external application and read audio from it. This thread also makes use of the MoH class object. In the MoH class destructor, we used pthread_cancel() to ask the thread to exit. Unfortunately, the code did not wait to ensure that the thread actually went away. What needed to be done is a pthread_join() to ensure that the thread fully cleans up before we proceed. By adding this one line, we resolve two significant problems: 1) Since the thread was never joined, it never fully goes away. So, on every reload of non-files mode MoH, an unused thread was sticking around. 2) There was a race condition here where the application monitoring thread could still try to access the MoH class, even though the thread executing the MoH reload has already destroyed it. (issue ASTERISK-14129) Reported by: jvandal (issue ASTERISK-14141) Reported by: axisinternet (issue ASTERISK-14203) Reported by: amorsen (issue AST-208) ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=201612 By: Digium Subversion (svnbot) 2009-06-18 10:36:14 Repository: asterisk Revision: 201613 _U branches/1.6.1/ U branches/1.6.1/res/res_musiconhold.c ------------------------------------------------------------------------ r201613 | russell | 2009-06-18 10:36:11 -0500 (Thu, 18 Jun 2009) | 43 lines Merged revisions 201610 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r201610 | russell | 2009-06-18 10:27:10 -0500 (Thu, 18 Jun 2009) | 36 lines Merged revisions 201600 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r201600 | russell | 2009-06-18 10:24:31 -0500 (Thu, 18 Jun 2009) | 29 lines Fix memory corruption and leakage related reloads of non files mode MoH classes. For Music on Hold classes that are not files mode, meaning that we are executing an application that will feed us audio data, we use a thread to monitor the external application and read audio from it. This thread also makes use of the MoH class object. In the MoH class destructor, we used pthread_cancel() to ask the thread to exit. Unfortunately, the code did not wait to ensure that the thread actually went away. What needed to be done is a pthread_join() to ensure that the thread fully cleans up before we proceed. By adding this one line, we resolve two significant problems: 1) Since the thread was never joined, it never fully goes away. So, on every reload of non-files mode MoH, an unused thread was sticking around. 2) There was a race condition here where the application monitoring thread could still try to access the MoH class, even though the thread executing the MoH reload has already destroyed it. (issue ASTERISK-14129) Reported by: jvandal (issue ASTERISK-14141) Reported by: axisinternet (issue ASTERISK-14203) Reported by: amorsen (issue AST-208) ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=201613 By: Digium Subversion (svnbot) 2009-06-18 10:40:19 Repository: asterisk Revision: 201614 _U branches/1.6.2/ U branches/1.6.2/res/res_musiconhold.c ------------------------------------------------------------------------ r201614 | russell | 2009-06-18 10:40:16 -0500 (Thu, 18 Jun 2009) | 43 lines Merged revisions 201610 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r201610 | russell | 2009-06-18 10:27:10 -0500 (Thu, 18 Jun 2009) | 36 lines Merged revisions 201600 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r201600 | russell | 2009-06-18 10:24:31 -0500 (Thu, 18 Jun 2009) | 29 lines Fix memory corruption and leakage related reloads of non files mode MoH classes. For Music on Hold classes that are not files mode, meaning that we are executing an application that will feed us audio data, we use a thread to monitor the external application and read audio from it. This thread also makes use of the MoH class object. In the MoH class destructor, we used pthread_cancel() to ask the thread to exit. Unfortunately, the code did not wait to ensure that the thread actually went away. What needed to be done is a pthread_join() to ensure that the thread fully cleans up before we proceed. By adding this one line, we resolve two significant problems: 1) Since the thread was never joined, it never fully goes away. So, on every reload of non-files mode MoH, an unused thread was sticking around. 2) There was a race condition here where the application monitoring thread could still try to access the MoH class, even though the thread executing the MoH reload has already destroyed it. (issue ASTERISK-14129) Reported by: jvandal (issue ASTERISK-14141) Reported by: axisinternet (issue ASTERISK-14203) Reported by: amorsen (issue AST-208) ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=201614 By: Russell Bryant (russell) 2009-06-19 11:10:57 If the problem still occurs, if anyone is able to make this happen with valgrind (see doc/valgrind.txt), that would be very helpful. By: David Brillert (aragon) 2009-06-23 15:37:02 russell I can still make the crash happen when in moh files mode. valgrind data is uploaded to https://issues.asterisk.org/view.php?id=15109 and a new backtrace after patching asterisk also https://issues.asterisk.org/view.php?id=15109#106764 I still cannot reproduce the actual crash in valgrind but hopeful that some of the uploaded valgrind data will be useful... By: Leif Madsen (lmadsen) 2009-07-13 10:28:43 Changed to Acknowledged as the info recently attached may be useful. Thanks! By: David Brillert (aragon) 2009-07-21 13:21:10 Testing SVN revision 206273 the moh files based segfaults are gone. Can anyone else test 206273 SVN and confirm? By: Russell Bryant (russell) 2009-08-25 14:50:33 I have posted a patch on issue 15109 which should address this issue. Please give it a try! https://issues.asterisk.org/view.php?id=15109 By: Digium Subversion (svnbot) 2009-09-01 15:39:17 Repository: asterisk-addons Revision: 1023 U branches/1.4/formats/format_mp3.c ------------------------------------------------------------------------ r1023 | russell | 2009-09-01 15:38:54 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk-addons?view=rev&revision=1023 By: Digium Subversion (svnbot) 2009-09-01 15:40:28 Repository: asterisk-addons Revision: 1024 U branches/1.6.0/formats/format_mp3.c ------------------------------------------------------------------------ r1024 | russell | 2009-09-01 15:40:16 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk-addons?view=rev&revision=1024 By: Digium Subversion (svnbot) 2009-09-01 15:42:36 Repository: asterisk-addons Revision: 1025 U branches/1.6.1/formats/format_mp3.c ------------------------------------------------------------------------ r1025 | russell | 2009-09-01 15:42:24 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk-addons?view=rev&revision=1025 By: Digium Subversion (svnbot) 2009-09-01 15:43:25 Repository: asterisk-addons Revision: 1026 U branches/1.6.2/formats/format_mp3.c ------------------------------------------------------------------------ r1026 | russell | 2009-09-01 15:43:13 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk-addons?view=rev&revision=1026 By: Digium Subversion (svnbot) 2009-09-01 15:45:09 Repository: asterisk Revision: 215212 U trunk/addons/format_mp3.c ------------------------------------------------------------------------ r215212 | russell | 2009-09-01 15:44:57 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=215212 By: Digium Subversion (svnbot) 2009-09-01 15:45:41 Repository: asterisk Revision: 215213 _U branches/1.6.2/ ------------------------------------------------------------------------ r215213 | russell | 2009-09-01 15:45:26 -0500 (Tue, 01 Sep 2009) | 51 lines Blocked revisions 215212 via svnmerge ........ r215212 | russell | 2009-09-01 15:44:13 -0500 (Tue, 01 Sep 2009) | 45 lines Fix memory corruption caused by format_mp3. format_mp3 claimed that it provided AST_FRIENDLY_OFFSET in frames returned by read(). However, it lied. This means that other parts of the code that attempted to make use of the offset buffer would end up corrupting the fields in the ast_filestream structure. This resulted in quite a few crashes due to unexpected values for fields in ast_filestream. This patch closes out quite a few bugs. However, some of these bugs have been open for a while and have been an area where more than one bug has been discussed. So with that said, anyone that is following one of the issues closed here, if you still have a problem, please open a new bug report for the specific problem you are still having. If you do, please ensure that the bug report is based on the newest version of Asterisk, and that this patch is applied if format_mp3 is in use. Thanks! (closes issue ASTERISK-14129) Reported by: jvandal Tested by: aragon, russell, zerohalo, marhbere, rgj (closes issue ASTERISK-14007) Reported by: aragon (closes issue ASTERISK-14141) Reported by: axisinternet (closes issue ASTERISK-14074) Reported by: maxnuv (closes issue ASTERISK-14374) Reported by: aragon (closes issue ASTERISK-14203) Reported by: amorsen Tested by: amorsen (closes issue ASTERISK-14718) Reported by: jensvb (closes issue ASTERISK-14673) Reported by: thom4fun (closes issue ASTERISK-14428) Reported by: marhbere ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=215213 | ||