[Home]

Summary:ASTERISK-14204: [patch] Asterisk crash when using ODBC to insert record on table that does not exist
Reporter:agjohnson (agjohnson)Labels:
Date Opened:2009-05-26 10:35:24Date Closed:2011-06-07 14:00:24
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Functions/func_odbc
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20090526__debug_issue15196__3.diff.txt
( 1) 20090526__debug_issue15196__5.diff.txt
( 2) 20090526__debug_issue15196.diff.txt
( 3) 20090528__issue15196.diff.txt
( 4) bt
( 5) bt-full
( 6) bt-full-05-28-09-1.txt
( 7) bt-full-latest-unixODBC.txt
( 8) console-output.txt
( 9) console-output-05-28-09-1.txt
(10) console-output-patch3.txt
(11) console-output-patch5.txt
(12) console-output-valgrind.txt
(13) core.15787.tar.gz
(14) db-queries.txt
(15) malloc_debug.txt
(16) refs
(17) refs-05-28-09-1
(18) refs-patch5
(19) refs-valgrind
(20) valgrind.txt
(21) valgrind-05-28-09-1
(22) valgrind-new.txt
Description:When using func_odbc to insert a record into a table that does not exist, asterisk will crash after attempting to reconnect to the database.  I have not tested this in other error scenarios.

MySQL Server version: mysql-server.x86_64 5.0.45-7.el5

unixODBC.i386       2.2.11-7.1
unixODBC-devel.i386 2.2.11-7.1

func_odbc.conf
[SET]
prefix=DB
dsn=mysql
writesql = INSERT INTO asterisk.CallVariables (uid,name,value) VALUES ('${SQL_ESC(${ARG1})}','${SQL_ESC(${VAL1})}','${SQL_ESC(${VAL2})}')
writesql += ON DUPLICATE KEY UPDATE value='${SQL_ESC(${VAL2})}'


Comments:By: agjohnson (agjohnson) 2009-05-26 10:49:59

This is not occurring in 1.6.1.0
By: Tilghman Lesher (tilghman) 2009-05-26 12:44:15

Additional gdb information needed:

p *obj
p *(obj->parent)
By: Tilghman Lesher (tilghman) 2009-05-26 12:55:05

Also, please apply this patch and get console.  When it crashes, please upload the file /tmp/refs.
By: agjohnson (agjohnson) 2009-05-26 12:57:01

(gdb) p *obj
$3 = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __kind = -1, __nusers = 0, {__spins = 0, __list = {__next = 0x0}}}, __size = '\0' <repeats 12 times>, "ÿÿÿÿ\000\000\000\000\000\000\000",
   __align = 0}, con = 0x0, parent = 0x0, last_used = {tv_sec = 1243351070, tv_usec = 853497}, used = 1, up = 0, tx = 0, txf = 0x0, list = {next = 0x0}}
(gdb)


(gdb) p *(obj->parent)
Cannot access memory at address 0x0
(gdb)
By: agjohnson (agjohnson) 2009-05-26 13:09:44

Added refs attachment
By: Tilghman Lesher (tilghman) 2009-05-26 13:46:14

Okay, that gives me a better idea as to what is happening, although the 'why' is still unclear.  Could you apply this new debugging patch, and run './configure --enable-dev-mode' before recompiling, please?  This should provide a backtrace into the code as to when the problem is actually occurring and why (instead of only after the fact, when the crash occurs).
By: agjohnson (agjohnson) 2009-05-26 14:07:35

There was a conflict with the refs patch, so I reverted back to -beta2 and applied the new patch.  Compiling now.
By: agjohnson (agjohnson) 2009-05-26 14:10:57

Compiler error:

  [CCi] res_odbc.i -> res_odbc.o
cc1: warnings being treated as errors
res_odbc.c: In function âodbc_release_obj2â:
res_odbc.c:1039: warning: passing argument 1 of â_ao2_unlink_debugâ from incompatible pointer type
res_odbc.c:1043: warning: implicit declaration of function âast_refâ
make[1]: *** [res_odbc.o] Error 1
make: *** [res] Error 2
By: Tilghman Lesher (tilghman) 2009-05-26 14:50:07

Oops, sorry.  New patch uploaded.
By: agjohnson (agjohnson) 2009-05-26 15:10:45

Attached console output from crash after patch.
By: Tilghman Lesher (tilghman) 2009-05-26 17:06:04

Okay, I THINK I've figured out where the errant line was.  I wasn't properly incrementing a refcount when storing an object in a transactional container, so when it was released, it was decrementing it one too many.  The latest patch includes a whole bunch more debugging, so if it crashes again, I'll need the /tmp/refs again.
By: agjohnson (agjohnson) 2009-05-26 17:28:26

Compile error:

res_odbc.c: In function ârelease_transactionâ:
res_odbc.c:319: error: too many arguments to function âodbc_release_obj2â
res_odbc.c: At top level:
res_odbc.c:981: error: conflicting types for âodbc_release_obj2â
res_odbc.c:145: error: previous declaration of âodbc_release_obj2â was here
cc1: warnings being treated as errors
res_odbc.c: In function âodbc_release_obj2â:
res_odbc.c:1051: warning: passing argument 3 of â_ao2_ref_debugâ discards qualifiers from pointer target type
res_odbc.c: In function âast_odbc_release_objâ:
res_odbc.c:1057: error: too few arguments to function âodbc_release_obj2â
make[1]: *** [res_odbc.o] Error 1
make: *** [res] Error 2
By: Tilghman Lesher (tilghman) 2009-05-26 18:23:12

Crud, I had fixed that, but forgot to update the patch.  Patch updated now.
By: agjohnson (agjohnson) 2009-05-27 09:00:42

New crash info uploaded
By: agjohnson (agjohnson) 2009-05-27 10:43:09

This crash is not happening if the table exists, but the query returns an error.
By: Tilghman Lesher (tilghman) 2009-05-27 16:25:50

That's rather weird.  There's nothing different that goes on to differentiate a table which exists from a table which does not exist.
By: Tilghman Lesher (tilghman) 2009-05-27 16:26:21

Maybe valgrind has a better idea of what's going on?  doc/valgrind.txt.
By: agjohnson (agjohnson) 2009-05-27 18:18:49

Other errors have resulted in a crash as well.  It is not just limited to tables not being there.  I will read the docs on valgrind.
By: Tilghman Lesher (tilghman) 2009-05-27 18:39:37

Right, but it's not actually crashing at that stage.  Instead, it's crashing at the point of a memory deallocation, which may point towards valgrind being helpful.
By: agjohnson (agjohnson) 2009-05-28 09:36:59

malloc_debug.txt was empty.  valgrind attached.
By: agjohnson (agjohnson) 2009-05-28 09:49:23

Asterisk did not crash, but I did get a lot of errors saying: Reference counts incorrect!  Unlinking database object from container as a safeguard.
By: Tilghman Lesher (tilghman) 2009-05-28 10:53:53

Okay, once again, I think I've found the fix.  It's a one-liner and should not conflict with the earlier debug patch (in other words, you can have either or both installed at once).
By: agjohnson (agjohnson) 2009-05-28 11:14:42

Still crashing.  I'm rebuilding asterisk with the original 1.6.2-beta2 sources and applying your patch.
By: agjohnson (agjohnson) 2009-05-28 11:41:04

Attached new console output and refs file.  There does not appear to be a backtrace.  Should I apply another patch to re-enable that?
By: agjohnson (agjohnson) 2009-05-28 12:39:17

bt full and new valgrind uploaded.
By: Tilghman Lesher (tilghman) 2009-05-28 13:14:41

Yay!  Now it's no longer an Asterisk crash, but a UnixODBC crash.  Just to make sure, try upgrading to the latest UnixODBC from source (as opposed to distribution package) as this usually solves crash issues like this.
By: agjohnson (agjohnson) 2009-05-28 15:58:33

I'm now using the latest unixODBC and MySQL drivers and Asterisk is still crashing.  I will post the files in just a moment.


By: agjohnson (agjohnson) 2009-05-28 16:24:24

This crash also occurs with the 5.1 version of the MySQL ODBC driver.
By: Tilghman Lesher (tilghman) 2009-05-28 17:06:08

Right, but it's still crashing in UnixODBC code.  All of the values passed to the routine seem to be reasonable, though.  The latest backtrace would seem to have good debugging information for the UnixODBC folks, so you may want to involve them at this point.  Their project leader, Nick Gorham, is very helpful and reasonable when it comes to possible bugs.
By: Leif Madsen (lmadsen) 2009-06-16 14:06:18

Closed as this appears to be an issue in the unixODBC library. Thanks!