[Home]

Summary:ASTERISK-18913: segfault in cdr_adaptive_odbc.c when database connection is interrupted
Reporter:Thomas Arimont (tomaso)Labels:
Date Opened:2011-11-23 03:01:56.000-0600Date Closed:2011-12-01 14:53:53.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:CDR/cdr_adaptive_odbc
Versions:1.8.5.0 Frequency of
Occurrence		Constant
Related
Issues:
Environment:rhel 6.1, distributed mysql database servers for cdr, queues_log,..., Attachments:( 0) bt_full_with_libodbc_debug_infos.txt
( 1) bt_full.log
( 2) cli.log
Description:If a call (calls) to a waitqueue is (are) made while the connection to the external cdr mysql database is interrupted, a segmentation fault can be observed in cdr_adaptive_odbc.c after a short time. Please see attached cli log and backtrace.

ODBC configuration (res_odbc.conf):

[odbc-voipstat101]
enabled => yes
dsn => voipstat101
pooling => no
pre-connect => yes
connect_timeout => 2
Comments:By: Thomas Arimont (tomaso) 2011-11-25 01:29:29.421-0600

The mysql-connector-odbc-5.1.5r1144-7 is used on RHEL 6.1. I will try to reproduce the segfault in my local test environment (the current report was from another test location) and get you the "bt full" backtrace along with debug symbols for the relevant libodbc function calls.
By: Thomas Arimont (tomaso) 2011-11-29 03:50:24.526-0600

Here comes the backtrace including the libodbc debug symbols. unixODBC-2.2.14-11.el6.x86_64 is active on the system. With the following odbc config we can reproduce this issue within minutes by generating calls to a call queue (QueueLog() is invoked) and disabling the mysql server's interface after a short while:

/etc/odbc.ini:
voipstat101]
Description         = MySQL RW connection to 'asterisk' database
Driver              = MySQL
Database            = ast_cdr
Server              = 10.54.100.5
User                = root
Password            = ****
Port                = 3306
ConnSettings         =
writetimeout         = 1


/etc/odbcinst.ini:
[MySQL]
Description     = ODBC for MySQL
Driver          = /usr/lib64/libmyodbc5.so
Setup           = /usr/lib64/libodbcmyS.so
FileUsage       = 1


/etc/asterisk/res_odbc.conf:
[odbc-voipstat101]
enabled => yes
dsn => voipstat101
pooling => no
pre-connect => yes
connect_timeout => 2


/etc/asterisk/cdr_adaptive_odbc.conf:
[adaptive-odbc-voipstat101]
connection=odbc-voipstat101
loguniqueid=yes
dispositionstring=yes
table=ast_cdr           ;"cdr" is default table name


/etc/asterisk/cdr_odbc.conf:
[global]
dsn=odbc-voipstat101
loguniqueid=yes
dispositionstring=yes
table=ast_cdr           ;"cdr" is default table name

By: Thomas Arimont (tomaso) 2011-11-29 09:15:53.747-0600

The additional gdb info below shows that the statement_handle is valid, but the connection pointer is NULL which explains this crash. Should this be fixed in unixODBC ?

{noformat}
(gdb) print *statement
$1 = {type = 0, next_class_list = 0x0, msg = '\000' <repeats 1023 times>, state = 0, connection = 0x0, driver_stmt = 0x0, hascols = 0,
 prepared = 0, interupted_func = 0, interupted_state = 0, bookmarks_on = 0, error = {sql_error_head = {error_count = 0, error_list_head = 0x0,
     error_list_tail = 0x0, internal_count = 0, internal_list_head = 0x0, internal_list_tail = 0x0}, sql_diag_head = {error_count = 0,
     error_list_head = 0x0, error_list_tail = 0x0, internal_count = 0, internal_list_head = 0x0, internal_list_tail = 0x0},
   owning_handle = 0x0, handle_type = 0, return_code = 0, header_set = 0, diag_cursor_row_count_ret = 0, diag_dynamic_function_ret = 0,
   diag_dynamic_function_code_ret = 0, diag_number_ret = 0, diag_row_count_ret = 0, diag_cursor_row_count = 0, diag_dynamic_function = {
     0 <repeats 128 times>}, diag_dynamic_function_code = 0, diag_number = 0, diag_row_count = 0}, metadata_id = 0, ipd = 0x0, apd = 0x0,
 ird = 0x0, ard = 0x0, implicit_ipd = 0x0, implicit_apd = 0x0, implicit_ird = 0x0, implicit_ard = 0x0, fetch_bm_ptr = 0x0, row_ct_ptr = 0x0,
 row_st_arr = 0x0, row_array_size = 0, valueptr = 0x0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
     __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}
(gdb)

{noformat}
By: Leif Madsen (lmadsen) 2011-11-29 13:44:01.623-0600

From the UnixODBC website:

* 19.Nov.2008 2.2.14 Released

Your version is from 2008. Please use the latest version from the UnixODBC team (which was released on November 28, 2011).
By: Thomas Arimont (tomaso) 2011-11-30 01:54:36.094-0600

Leif,
unixODBC-2.2.14-11 is the current RHEL6.1 or Scientific Linux 6.1 library (Patch-Level 11 !!!).
Bit I even tried it with the recent sources from unixODBC (2.3.1) and even with the current mysql-connector mysql-connector-odbc from MySQL (5.1.9).
The result remains the same.
By: Tilghman Lesher (tilghman) 2011-12-01 14:51:35.818-0600

It most likely needs to be fixed in the MySQL ODBC connector, not even UnixODBC.
By: Leif Madsen (lmadsen) 2011-12-01 14:53:47.326-0600

I'm going to close this as it appears to be outside of Asterisk.

Per developer on IRC:

<Corydon76-home> If it occurs consistently when the database is down, then it's outside of Asterisk's purview
<Corydon76-home> I can say that pretty confidently, because ODBC is responsible for the connection, not Asterisk.
<Corydon76-home> (and given that this is MySQL, it's pretty much always the MySQL ODBC connector that is at fault)