[Home]

Summary:ASTERISK-11694: segfault when using DETECT_DEADLOCKS flag
Reporter:Clod Patry (junky)Labels:
Date Opened:2008-03-21 09:09:39Date Closed:2008-03-21 10:19:38
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:I've got that random crash:

(gdb) bt
#0  0x00000000004dcac5 in ast_mark_lock_acquired (lock_addr=0x2a98c68470) at utils.c:644
#1  0x0000002a98b5a860 in __ast_pthread_mutex_lock (filename=0x2a98b63d8b "chan_agent.c", lineno=2579, func=0x2a98b666e0 "agent_devicestate", mutex_name=0x2a98b652a3 "&(&agents)->lock", t=0x2a98c68470)
   at /usr/src/ah_ast_1.4/include/asterisk/lock.h:374
#2  0x0000002a98b633f9 in agent_devicestate (data=0x40076fe6) at chan_agent.c:2579
#3  0x000000000045efd7 in ast_device_state (device=0x2389958 "Agent/1029") at devicestate.c:170
#4  0x000000000045fa7d in do_state_change (device=0x2389958 "Agent/1029") at devicestate.c:285
ASTERISK-1  0x000000000045ff2e in do_devstate_changes (data=0x0) at devicestate.c:369
ASTERISK-2  0x00000000004dd397 in dummy_start (data=0x69c840) at utils.c:865
ASTERISK-3  0x0000003085e06137 in start_thread () from /lib64/tls/libpthread.so.0
ASTERISK-4  0x00000030857c7543 in clone () from /lib64/tls/libc.so.6
(gdb) bt full
#0  0x00000000004dcac5 in ast_mark_lock_acquired (lock_addr=0x2a98c68470) at utils.c:644
       lock_info = (struct thr_lock_info *) 0x6cada0
#1  0x0000002a98b5a860 in __ast_pthread_mutex_lock (filename=0x2a98b63d8b "chan_agent.c", lineno=2579, func=0x2a98b666e0 "agent_devicestate", mutex_name=0x2a98b652a3 "&(&agents)->lock", t=0x2a98c68470)
   at /usr/src/ah_ast_1.4/include/asterisk/lock.h:374
       res = 0
       canlog = 1
       __PRETTY_FUNCTION__ = "__ast_pthread_mutex_lock"
#2  0x0000002a98b633f9 in agent_devicestate (data=0x40076fe6) at chan_agent.c:2579
       p = (struct agent_pvt *) 0x8f14b0
       s = 0x40076fe6 "1029"
       groupmatch = 0
       groupoff = 0
       waitforagent = 0
       res = 4
       __PRETTY_FUNCTION__ = "agent_devicestate"
#3  0x000000000045efd7 in ast_device_state (device=0x2389958 "Agent/1029") at devicestate.c:170
       buf = 0x40076fe6 "1029"
       number = 0x40076fe6 "1029"
       chan_tech = (const struct ast_channel_tech *) 0x2a98c67720
       res = 0
       tech = 0x40076fe0 "Agent"
       provider = 0x0
       __PRETTY_FUNCTION__ = "ast_device_state"
#4  0x000000000045fa7d in do_state_change (device=0x2389958 "Agent/1029") at devicestate.c:285
       state = 0
       devcb = (struct devstate_cb *) 0x51b633
       __PRETTY_FUNCTION__ = "do_state_change"
ASTERISK-1  0x000000000045ff2e in do_devstate_changes (data=0x0) at devicestate.c:369
       cur = (struct state_change *) 0x2389950
       __PRETTY_FUNCTION__ = "do_devstate_changes"
ASTERISK-2  0x00000000004dd397 in dummy_start (data=0x69c840) at utils.c:865
       _buffer = {__routine = 0x428a68 <ast_unregister_thread>, __arg = 0x40077960, __canceltype = 0, __prev = 0x0}
       ret = (void *) 0x0
       a = {start_routine = 0x45fe79 <do_devstate_changes>, data = 0x0, name = 0x69c880 "do_devstate_changes  started at [  386] devicestate.c ast_device_state_engine_init()"}
       lock_info = (struct thr_lock_info *) 0x6cada0
       mutex_attr = {__mutexkind = 1}
ASTERISK-3  0x0000003085e06137 in start_thread () from /lib64/tls/libpthread.so.0
No symbol table info available.
ASTERISK-4  0x00000030857c7543 in clone () from /lib64/tls/libc.so.6
No symbol table info available.


****** ADDITIONAL INFORMATION ******

(gdb) p lock_info->locks[lock_info->num_locks - 1]
$1 = {file = 0x6b73697265747361 <Address 0x6b73697265747361 out of bounds>, line_num = 1629512494, func = 0x656b636f73656b61 <Address 0x656b636f73656b61 out of bounds>,
 lock_name = 0x292874 <Address 0x292874 out of bounds>, lock_addr = 0xe51, times_locked = 1074231648, type = AST_MUTEX, pending = 0}


I dont see any commit to lock.h, so i think this is still an issue with latest Rev.
Comments:By: Mark Michelson (mmichelson) 2008-03-21 09:17:01

JunK-Y:

Could you please try the latest version? There have been a couple of changes to utils.c which fixed some logic flaws in this section of code. If it's still a problem, please let us know.
By: Mark Michelson (mmichelson) 2008-03-21 09:18:07

Oh, and just as a sanity check, if you still have the core file, could you run the following command in gdb:

p lock_info->num_locks

Thanks.
By: Clod Patry (junky) 2008-03-21 09:20:30

(gdb) p lock_info->num_locks
$1 = 0
(gdb)


and to push it a little bit:
(gdb) p *lock_info
$2 = {thread_id = 1074231648, thread_name = 0x6cbbf0 "do_devstate_changes  started at [  386] devicestate.c ast_device_state_engine_init()", locks = {{file = 0x522173 "logger.c", line_num = 740,
     func = 0x523abf "ast_log", lock_name = 0x5223a2 "&(&logchannels)->lock", lock_addr = 0x663ed0, times_locked = 1, type = AST_MUTEX, pending = 0}, {file = 0x515454 "channel.c", line_num = 1065,
     func = 0x516cd0 "channel_find_locked", lock_name = 0x516cfe "&c->lock", lock_addr = 0x913898, times_locked = 0, type = AST_MUTEX, pending = -1}, {file = 0x0, line_num = 0, func = 0x0,
     lock_name = 0x0, lock_addr = 0x0, times_locked = 0, type = AST_MUTEX, pending = 0} <repeats 62 times>}, num_locks = 0, lock = {__m_reserved = 1, __m_count = 1, __m_owner = 0x100000f38, __m_kind = 1,
   __m_lock = {__status = 0, __spinlock = 0}}, entry = {next = 0x6cbe10}}
By: Mark Michelson (mmichelson) 2008-03-21 10:16:18

So if lock_info->num_lcoks is 0, then it makes sense that there would be a crash since out-of-bounds memory is referenced. The thing is, it should be impossible to call ast_mark_lock_acquired with lock_info->num_locks < 1.
By: Clod Patry (junky) 2008-03-21 10:19:37

seems fixed with revision: 110474.