[Home]

Summary:ASTERISK-10614: Asterisk 1.4.13 stock segfault on pthread_mutex_lock
Reporter:Leif Jackson (ljackson)Labels:
Date Opened:2007-10-24 16:48:19Date Closed:2011-06-07 14:00:36
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:after doing some config changes and issuing a iax2 reload and extensions reload, it happend on the 3rd iax2 reload, asterisk segfaulted I ran gdb on the core and this is the BT:

Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
#0  0x003ca2a0 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) where
#0  0x003ca2a0 in pthread_mutex_lock () from /lib/libpthread.so.0
#1  0x0376c01b in iax2_poke_peer (peer=0x9d296d8, heldcall=0)
   at /root/asterisk-dist/asterisk-1.4.13/include/asterisk/lock.h:610
#2  0x0378042c in iax2_process_thread (data=0x9ce6150) at chan_iax2.c:8259
#3  0x080f643b in dummy_start (data=0x9cde360) at utils.c:806
#4  0x003c82db in start_thread () from /lib/libpthread.so.0
ASTERISK-1  0x0032212e in clone () from /lib/libc.so.6




****** ADDITIONAL INFORMATION ******

CentOS 5 with all updates.
Comments:By: Volnikov Ivan (ivan) 2007-10-25 04:41:01

ljackson -
Please compile your Asterisk with DEBUG_THREADS, DONT_OPTIMIZE, MALLOC_DEBUG and LOADABLE_MODULES compiler flags and place "bt full" log.


By: Leif Jackson (ljackson) 2007-10-25 07:33:54

I have done so. Will let you know if I can reproduce with these debuging options turned on. How much does this effect performance on calls?

Thanks,
Leif
By: Leif Jackson (ljackson) 2007-10-25 09:09:53

the options DEBUG_THREADS and MALLOC_DEBUG made the audio sound very bad I had to take them back out. I will still post a backtrace if I get a crash.

I personally believe this issue might be with how the thread mutexs are handled or a race condition between when a iax reload is issued and the iax2_peer_poke happens I believe that due to these both happening at the same time, I think this caused the crash I saw.

Thanks,
Leif
By: Russell Bryant (russell) 2007-10-30 08:31:19

Can you tell me some more about about IAX setup?  Are you using "realtime" for configuration?  If so, what realtime options do you have set in iax.conf?  Also, how many peers do you have defined?
By: Digium Subversion (svnbot) 2007-10-30 10:51:02

Repository: asterisk
Revision: 87569

A   team/russell/iax2_poke_fix/

------------------------------------------------------------------------
r87569 | russell | 2007-10-30 10:51:00 -0500 (Tue, 30 Oct 2007) | 9 lines

Create a branch for a small project I'm working on.  There is a race condition
related to the handling of POKEing peers.  Essentially, a reference to a peer
is held by the scheduler when there are pending callbacks, but the reference
count doesn't reflect it.  So, it is possible for a peer to hit a reference
count of zero and have its destructor begin to be called at the same time that
the scheduler thread runs a POKE related callback.  If that happens, a crash
will likely occur.
(related to issues ASTERISK-10614, ASTERISK-10626)

------------------------------------------------------------------------
By: Digium Subversion (svnbot) 2007-10-30 16:18:06

Repository: asterisk
Revision: 87686

U   branches/1.4/channels/chan_iax2.c

------------------------------------------------------------------------
r87686 | russell | 2007-10-30 16:18:06 -0500 (Tue, 30 Oct 2007) | 11 lines

Merge the changes from team/russell/iax2_poke_fix and iax2-poke-fix-trunk

There was a race condition related to the handling of POKEing peers.  Essentially,
a reference to a peer is held by the scheduler when there are pending callbacks,
but the reference count didn't reflect it.  So, it was possible for a peer to hit
a reference count of zero and have its destructor begin to be called at the same
time that the scheduler thread ran a POKE related callback.  If that happened,
a crash would likely occur.

(closes issue ASTERISK-10614, closes issue ASTERISK-10626)

------------------------------------------------------------------------
By: Digium Subversion (svnbot) 2007-10-30 16:21:08

Repository: asterisk
Revision: 87687

_U  trunk/
U   trunk/channels/chan_iax2.c

------------------------------------------------------------------------
r87687 | russell | 2007-10-30 16:21:07 -0500 (Tue, 30 Oct 2007) | 19 lines

Merged revisions 87686 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r87686 | russell | 2007-10-30 16:19:09 -0500 (Tue, 30 Oct 2007) | 11 lines

Merge the changes from team/russell/iax2_poke_fix and iax2-poke-fix-trunk

There was a race condition related to the handling of POKEing peers.  Essentially,
a reference to a peer is held by the scheduler when there are pending callbacks,
but the reference count didn't reflect it.  So, it was possible for a peer to hit
a reference count of zero and have its destructor begin to be called at the same
time that the scheduler thread ran a POKE related callback.  If that happened,
a crash would likely occur.

(closes issue ASTERISK-10614, closes issue ASTERISK-10626)

........

------------------------------------------------------------------------
By: Leif Jackson (ljackson) 2007-11-06 13:37:43.000-0600

Still hapinging with the svn version with the fixes. this site using mysql realtime:

---------------------------------------------

Asterisk SVN-branch-1.4-r87852, Copyright (C) 1999 - 2007 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
 == Parsing '/etc/asterisk/asterisk.conf': Found
 == Parsing '/etc/asterisk/extconfig.conf': Found
 == Binding iaxusers to mysql/asterisk/iax_peers
 == Binding iaxpeers to mysql/asterisk/iax_peers
 == Binding sipusers to mysql/asterisk/sip_peers
 == Binding sippeers to mysql/asterisk/sip_peers
 == Binding extensions to mysql/asterisk/pbx_extensions
Connected to Asterisk SVN-branch-1.4-r87852 currently running on custpbx00 (pid = 7812)
Verbosity is at least 3

-----------------------

Program terminated with signal 11, Segmentation fault.
#0  0x080eb8f5 in ast_sched_del (con=0x82628b8, id=1148) at sched.c:286
286                     CRASH;
(gdb) where
#0  0x080eb8f5 in ast_sched_del (con=0x82628b8, id=1148) at sched.c:286
#1  0x00bb4bf1 in iax2_destroy_helper (pvt=0xb79162b8) at chan_iax2.c:1913
#2  0x00bb4f2c in iax2_destroy (callno=2) at chan_iax2.c:1980
#3  0x00bd4ea3 in __iax2_poke_noanswer (data=0x82644d0) at chan_iax2.c:8513
#4  0x00bd4041 in iax2_process_thread (data=0x82670e8) at chan_iax2.c:8299
ASTERISK-1  0x080fa3c0 in dummy_start (data=0x8263f70) at utils.c:843
ASTERISK-2  0x003c82db in start_thread () from /lib/libpthread.so.0
ASTERISK-3  0x0032212e in clone () from /lib/libc.so.6
(gdb) bt full
#0  0x080eb8f5 in ast_sched_del (con=0x82628b8, id=1148) at sched.c:286
       s = (struct sched *) 0x0
       __PRETTY_FUNCTION__ = "ast_sched_del"
#1  0x00bb4bf1 in iax2_destroy_helper (pvt=0xb79162b8) at chan_iax2.c:1913
No locals.
#2  0x00bb4f2c in iax2_destroy (callno=2) at chan_iax2.c:1980
       pvt = (struct chan_iax2_pvt *) 0xb79162b8
       cur = (struct iax_frame *) 0xb7cd2b90
       owner = (struct ast_channel *) 0x0
       __PRETTY_FUNCTION__ = "iax2_destroy"
#3  0x00bd4ea3 in __iax2_poke_noanswer (data=0x82644d0) at chan_iax2.c:8513
       peer = (struct iax2_peer *) 0x82644d0
       __PRETTY_FUNCTION__ = "__iax2_poke_noanswer"
#4  0x00bd4041 in iax2_process_thread (data=0x82670e8) at chan_iax2.c:8299
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {12503980, 0, -1211290736,
       -1211292840, -551423833, 1750200032}, __mask_was_saved = 0}}, __pad = {0xb7cd2370,
   0x0, 0xc, 0x0}}
       __cancel_routine = (void (*)(void *)) 0xbd3b5e <iax2_process_thread_cleanup>
       __cancel_arg = (void *) 0x82670e8
       not_first_call = 0
       thread = (struct iax2_thread *) 0x82670e8
       tv = {tv_sec = -1211292904, tv_usec = 3979120}
       ts = {tv_sec = 0, tv_nsec = -1211290736}
       put_into_idle = 1
ASTERISK-1  0x080fa3c0 in dummy_start (data=0x8263f70) at utils.c:843
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {4026356, 0, -1211290736,
       -1211292728, -551423561, 1625518407}, __mask_was_saved = 0}}, __pad = {0xb7cd2480,
   0x0, 0x0, 0x0}}
       __cancel_routine = (void (*)(void *)) 0x8068c45 <ast_unregister_thread>
       __cancel_arg = (void *) 0xb7cd2b90
       not_first_call = 0
       ret = (void *) 0x0
       a = {start_routine = 0xbd3bd8 <iax2_process_thread>, data = 0x82670e8,
 name = 0x8264110 "iax2_process_thread  started at [ 8801] chan_iax2.c start_network_thread()"}
ASTERISK-2  0x003c82db in start_thread () from /lib/libpthread.so.0
No symbol table info available.
ASTERISK-3  0x0032212e in clone () from /lib/libc.so.6
No symbol table info available.
By: Russell Bryant (russell) 2007-11-06 14:35:19.000-0600

Don't run with DO_CRASH turned on ...