|Summary:||ASTERISK-19889: Asterisk crashes due to memory corruption|
|Reporter:||Scottie Winn (nhutasterisk)||Labels:|
|Date Opened:||2012-05-20 15:15:37||Date Closed:||2012-09-19 20:27:49|
|Environment:||Centos 5.8 Final||Attachments:||( 0) backtrace6.txt|
( 1) valgrind.txt
( 2) valgrind10_4.txt
( 3) valgrind10_4-leak-check-full.txt
|Description:||Seems to be at the manager API|
|Comments:||By: Scottie Winn (nhutasterisk) 2012-05-20 15:20:25.673-0500|
By: Matt Jordan (mjordan) 2012-05-21 08:08:40.412-0500
Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks!
By: Scottie Winn (nhutasterisk) 2012-05-21 21:59:28.490-0500
The backtrace6.txt was for 10.4
The valgrind.txt is for 10.5.0-rc1. Please let me know if you want me to reverse back to 10.4. Thanks
By: Richard Mudgett (rmudgett) 2012-05-22 10:41:33.242-0500
FYI You need to send the issue back otherwise it could fall off the radar waiting for you to respond.
By: Scottie Winn (nhutasterisk) 2012-05-22 11:24:08.922-0500
I will produce valgrind file for asterisk 10.4 tonight and upload it. Thanks
By: Scottie Winn (nhutasterisk) 2012-05-22 20:24:40.903-0500
These two files were generated under 10.4. Thanks
By: Rusty Newton (rnewton) 2012-05-23 17:13:02.194-0500
Scottie, thanks for the data. Can you additionally provide a detailed description of how the system is used (whats happening on the system when it crashes? queues with lots of inbound calls? AMI?, AGI?, what calls are being made? database connections? etc) and a full log with at least DEBUG 5 up to the time of the crash.
By: Scottie Winn (nhutasterisk) 2012-05-23 19:36:16.201-0500
Hi Rusty! The crashes happened even though I eliminated AMI connections. There were about 40 queues and each of them had less than 5 callers. We didn't have any crashes when we're using 1.6.2.X. I looked at the log 5 minutes before the crash and the only database connections were to astdb.sqlite and there were no AGI called. I reversed back to 220.127.116.11 last night and so far asterisk hasn't crashed yet. Usually asterisk would crash at least once by now when we're using 10.4 or 10.5-rc1. Most of the time when asterisk crashed, it didn't leave any segfault messages in the message log, but it generated core files in the tmp directory. Please let me know if you still want me to upload the log file for asterisk. Thanks.
By: Matt Jordan (mjordan) 2012-05-24 07:33:45.052-0500
I know you've moved back to 1.6.2.X - for any of the valgrind logs you uploaded, did any capture the output from an instance of Asterisk that crashed?
While there are a number of warnings in those outputs, none appear to be indicative of something that would cause the problems you had.
By: Scottie Winn (nhutasterisk) 2012-05-24 10:23:02.670-0500
All valgrind10_4 files were captured from Asterisk 10.4 and valgrind.txt were from 10.5-rc1. While running valgrind on Asterisk 10.4, Asterisk didn't crash because I ran it for a short period of time. It's on a production server so I couldn't afford to leave it running for long. We are on day 2 of 18.104.22.168 and haven't crashed yet. Please let me know if you need me to do anything more. Thanks.
By: Rusty Newton (rnewton) 2012-05-25 09:54:33.915-0500
Scottie, as Matt Jordan said "While there are a number of warnings in those outputs, none appear to be indicative of something that would cause the problems you had."
You'll need to get Valgrind output during the crash for a developer to be able to find anything useful in the output. Otherwise we won't have a way to investigate this issue, unless you can provide exact instructions for reproduction.
By: Matt Jordan (mjordan) 2012-06-21 12:38:59.150-0500
If you're using app_voicemail, do you mind trying the patch on ASTERISK-19923? A memory corruption has been identified in that application, and the patch on that issue has fixed it for some users experiencing memory corruption crashes.
The patch was originally written for 1.8, but - at least for me - applied cleanly to a fresh checkout of Asterisk 10 as well.
By: Rusty Newton (rnewton) 2012-07-23 19:03:37.709-0500
Scottie, have you had a chance to try this patch? Are you still experiencing the issue?
By: Scottie Winn (nhutasterisk) 2012-08-17 10:02:01.892-0500
I will install 10.7 on our production server and report if the bug is still there.
By: Rusty Newton (rnewton) 2012-09-19 20:27:40.848-0500
Suspended due to lack of activity. Please request a bug marshal in #asterisk-bugs on the IRC network irc.freenode.net to reopen the issue should you have the additional information requested. Further information can be found at https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines