Summary:	ASTERISK-19890: Crash occurs when using SIP realtime with SQL Server database, if a SIP client is un-registered
Reporter:	LiuYan刘研 (lovetide)	Labels:
Date Opened:	2012-05-21 03:11:19	Date Closed:	2017-12-20 09:00:17.000-0600
Priority:	Critical	Regression?
Status:	Closed/Complete	Components:	Resources/res_odbc
Versions:	1.8.12.0	Frequency of Occurrence	Constant
Related Issues:
Environment:	CentOS 5.8 x64 SQL Server 2000 SP4 / SQL Server 2008 R2 Express SP1 (running on Windows Server 2003)	Attachments:	( 0) 2012-06-27_1530_ServerCrashScreenRecordSIP_Realtime+SQLServer.swf.7z ( 1) backtrace_2012-06-27_1_core.26212.txt ( 2) backtrace_2012-06-27_2_core.26121.txt ( 3) backtrace_2012-06-27_3_core.26033.txt ( 4) backtrace_server_crash,_SIP_realtime_+_SQL_Server.txt ( 5) extconfig.conf ( 6) extensions.conf ( 7) odbc.ini ( 8) odbcinst.ini ( 9) res_odbc.conf (10) sip.conf (11) valgrind_2012-06-27.txt (12) valgrind_SIP_realtime_+_SQL_Server.zip
Description:	When using SIP realtime with SQL Server database (I tried both SQL Server 2000 SP4 and SQL Server 2008 R2 Express SP1), if SIP client unregistered, asterisk server will crash. If changed to PostgreSQL server, then all works fine.
Comments:	By: LiuYan刘研 (lovetide) 2012-05-21 03:12:17.533-0500 backtrace of one crash dump By: Rusty Newton (rnewton) 2012-05-23 17:20:40.431-0500 LiuYan, your backtrace has optimizations that prevent it from being useful to a developer. Please read through : https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace and see the section on "Verify your backtraces". You'll need to make sure Asterisk is compiled with the DONT_OPTIMIZE flag, as it mentions. Please also provide res_odbc.conf, extensions.conf and sip.conf, but of course sanitize any credentials. Please attach these as text files to the issue. By: LiuYan刘研 (lovetide) 2012-05-23 21:11:17.516-0500 Here're my config files. About the backtrace file: I just deleted the original backtrace.txt file. I'll upload new backtrace as soon as possible (I installed asterisk via yum, not compiled it by myself, I'll try compile asterisk...) By: LiuYan刘研 (lovetide) 2012-05-24 01:07:36.804-0500 new backtrace from asterisk which compiled with {{DONT_OPTIMIZE}} flag. By: LiuYan刘研 (lovetide) 2012-05-24 20:25:58.746-0500 new backtrace and config files uploaded By: Rusty Newton (rnewton) 2012-05-25 09:50:36.713-0500 LiuYan, thanks for all the data. The issue is acknowledged and has been moved into our queue. By: Rusty Newton (rnewton) 2012-06-04 09:11:43.123-0500 LiuYan,  if possible, please also provide valgrind output, as this issue may involve memory corruption and that output will be necessary to look into it:  https://wiki.asterisk.org/wiki/display/AST/Valgrind By: LiuYan刘研 (lovetide) 2012-06-05 02:36:47.492-0500 valgrind output file uploaded By: Matt Jordan (mjordan) 2012-06-25 09:25:51.009-0500 LiuYan: There is a suspected memory corruption in app_voicemail.  Can you check if the patch on ASTERISK-19923 resolves this issue for you? Thanks Matt By: LiuYan刘研 (lovetide) 2012-06-26 04:54:11.542-0500 I applied the patch, but the crash still occured. That patch is for voice mail, in my backtrace.txt, I didn't found anything about voice mail. By: Matt Jordan (mjordan) 2012-06-26 08:30:36.575-0500 Memory corruptions can cause memory allocation/deallocation crashes elsewhere in the code base.  It was worth checking to see if this resolved the issue for you. Three questions/follow-ups: 1) When you migrated to using a PostgreSQL server, did you change anything else other then what server the res_config_odbc pointed to? 2) Are backtraces from the other crashes occurring in the same location, or is the location different? 3) The file you uploaded as the valgrind output is not, in fact, the output from valgrind, but rather some CLI output from Asterisk that was produced at the same time as the valgrind output.  If using the instructions from the wiki, the valgrind output should be redirected to a separate file, which is the file that was requested.  As this appears to be a memory corruption, the valgrind output will be essential in debugging this problem. Note that valgrind may not report the problem unless Asterisk runs into the same scenario that caused the initial crash - so you may need to run for some time and produce the same circumstances that caused the initial crash in order for valgrind to report the problem. By: LiuYan刘研 (lovetide) 2012-06-27 05:08:14.188-0500 I've recorded screen to a .swf file (I paused recording for a while at 04:29, because valgrind make asterisk start very slow). (i) The asterisk shows in screen record had already applied the patch from [ASTERISK-19923|https://issues.asterisk.org/jira/browse/ASTERISK-19923] 1. I just change the realtime setting in {{extconfig.conf}}, then {{module reload extconfig}}, nothing else changed. (see 01:40 at the screen record) 2. I can't analyze the backtrace file, so I uploaded 3 backtrace files (also they're for 3 crashes in in the screen record). So far I can see in the 3 backtrace files is: The first 5 callstack items which revolved in asterisk in these 3 files are same. {code} ------------------------------------ odbc_obj_connect             at res_odbc.c:1544 ast_odbc_sanity_check        at res_odbc.c:760 ast_odbc_prepare_and_execute at res_odbc.c:663 update_odbc                  at res_config_odbc.c:543 ast_update_realtime          at config.c:2500 ------------------------------------ {code} Does this mean crashes occurred in the same location ? 3. I'm not sure why the first valgrind file contains console output, i use {{valgrind --suppressions=contrib/valgrind.supp --log-fd=9 /usr/local/sbin/asterisk -gcvvvvvddddd 9 > ~/asterisk-1.8.12-crash-valgrind.txt}} command to produced that valgrind file. I've uploaded a new {{valgrind.txt}} file, which is the one you can see in the screen record. I use the following command to produce it: {{valgrind --suppressions=/root/asterisk/asterisk-1.8.12.0/contrib/valgrind.supp --log-fd=9 /usr/local/sbin/asterisk -vvvvcg 9>valgrind.txt}} By: Joshua C. Colp (jcolp) 2017-12-19 05:51:44.368-0600 There's been considerable changes and fixes since this issue was filed. Are you still experiencing it under a supported version of Asterisk? By: LiuYan刘研 (lovetide) 2017-12-20 08:37:22.402-0600 I moved to asterisk-11 (CentOS 6) then asterisk-13 (Debian unstable). Currently I'm running Asterisk 13.18.3 on Debian unstable, I didn't encountered this issue anymore.