[Home]

Summary:ASTERISK-14314: Asterisk crash in ast_channel functions with dumps
Reporter:Kevin Scott Adams (nivek)Labels:
Date Opened:2009-06-15 07:01:47Date Closed:2011-06-07 14:00:21
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Channels/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:Over the past week we have had 3 crashes on one asterisk server that is an ACD server in the ast_channel functions (so it seems from the dumps).  A separate IVR box is running 1.6.0.9 also and has not crashed.  OpenSER's kamailio is also in the mix running as the proxy for most SIP traffic in the IVR/ACD system.  Dump information is supplied below.

The system has been running fine on 1.6.0.6 and 1.6.0.9 since about the first of April without issues.

The first dump has happened twice.

The second dump only once.

Running on CentOS 5.3 Kernel 2.6.18-128.1.10.el5PAE #1 SMP as a KVM under Proxmox VE 1.1 and recently upgraded (last week) to 1.3 with 2 processors, 4GB RAM and 80Gb HD space.

No Telephony cards are installed on the system (i.e.- DAHDI)



****** ADDITIONAL INFORMATION ******

<<<FIRST CRASH DUMP>>>

#0  0x00941402 in __kernel_vsyscall ()
#1  0x00992d80 in raise () from /lib/libc.so.6
#2  0x00994691 in abort () from /lib/libc.so.6
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
#4  0x009d30f1 in _int_free () from /lib/libc.so.6
ASTERISK-1  0x009d6bc0 in free () from /lib/libc.so.6
ASTERISK-2  0x08090d41 in ast_channel_free (chan=0xb68e94d8) at channel.c:1311
ASTERISK-3  0x08092013 in ast_hangup (chan=0xb68e94d8) at channel.c:1685
ASTERISK-4  0x08102021 in __ast_pbx_run (c=0xb68e94d8, args=0x0) at pbx.c:3882
ASTERISK-5  0x08102363 in pbx_thread (data=0xb68e94d8) at pbx.c:3974
ASTERISK-6 0x081502d0 in dummy_start (data=0xb689b208) at utils.c:861
ASTERISK-7 0x00ae449b in start_thread () from /lib/libpthread.so.0
ASTERISK-8 0x00a3b42e in clone () from /lib/libc.so.6




(gdb) bt full
#0  0x00941402 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00992d80 in raise () from /lib/libc.so.6
No symbol table info available.
#2  0x00994691 in abort () from /lib/libc.so.6
No symbol table info available.
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
No symbol table info available.
#4  0x009d30f1 in _int_free () from /lib/libc.so.6
No symbol table info available.
ASTERISK-1  0x009d6bc0 in free () from /lib/libc.so.6
No symbol table info available.
ASTERISK-2  0x08090d41 in ast_channel_free (chan=0xb68e94d8) at channel.c:1311
       fd = 0
       vardata = (struct ast_var_t *) 0x9886978
       f = (struct ast_frame *) 0x4a35c784
       headp = (struct varshead *) 0xb68e98e4
       datastore = (struct ast_datastore *) 0x0
       name = "\001\000\000\000p\227\216¶ \227\216¶Xð¤¶ý×\t\bT!\030\b|\006\000\000k8\030\b¢U\030\bX\225\216¶\024\001\000\0004ü\027\bbù\027\b¨<\033\b\000\000\000\000°ï¤¶\000\000\000\000\204Ç5J]\002\017\000\000\000\000"
       dashptr = 0xb6a4efb0 " \227\216¶\220û¤¶Xð¤¶A\r\t\bxô\202¶T!\030\b\036\005"
       __PRETTY_FUNCTION__ = "ast_channel_free"
ASTERISK-3  0x08092013 in ast_hangup (chan=0xb68e94d8) at channel.c:1685
       res = 0
       __PRETTY_FUNCTION__ = "ast_hangup"
ASTERISK-4  0x08102021 in __ast_pbx_run (c=0xb68e94d8, args=0x0) at pbx.c:3882
       found = 0
       res = -1
       autoloopflag = 0
       error = 1
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
ASTERISK-5  0x08102363 in pbx_thread (data=0xb68e94d8) at pbx.c:3974
       c = (struct ast_channel *) 0xb68e94d8
ASTERISK-6 0x081502d0 in dummy_start (data=0xb689b208) at utils.c:861
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {11481076, 0, -1230701680, -1230703688, 1944468800,
       -849883186}, __mask_was_saved = 0}}, __pad = {0xb6a4f470, 0x0, 0x98b53d8, 0x9871e98}}
       __cancel_routine = (void (*)(void *)) 0x80720a1 <ast_unregister_thread>
       __cancel_arg = (void *) 0xb6a4fb90
       not_first_call = 0
       ret = (void *) 0xb6a4f3b4
       a = {start_routine = 0x8102344 <pbx_thread>, data = 0xb68e94d8,
 name = 0xb68390e8 "pbx_thread", ' ' <repeats 11 times>, "started at [ 3995] pbx.c ast_pbx_start()"}
       lock_info = (struct thr_lock_info *) 0x98b53d8
       mutex_attr = {__size = "\001\000\000", __align = 1}
ASTERISK-7 0x00ae449b in start_thread () from /lib/libpthread.so.0
No symbol table info available.
ASTERISK-8 0x00a3b42e in clone () from /lib/libc.so.6
No symbol table info available.



Thread 1 (process 10874):
#0  0x00941402 in __kernel_vsyscall ()
#1  0x00992d80 in raise () from /lib/libc.so.6
#2  0x00994691 in abort () from /lib/libc.so.6
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
#4  0x009d30f1 in _int_free () from /lib/libc.so.6
ASTERISK-1  0x009d6bc0 in free () from /lib/libc.so.6
ASTERISK-2  0x08090d41 in ast_channel_free (chan=0xb68e94d8) at channel.c:1311
ASTERISK-3  0x08092013 in ast_hangup (chan=0xb68e94d8) at channel.c:1685
ASTERISK-4  0x08102021 in __ast_pbx_run (c=0xb68e94d8, args=0x0) at pbx.c:3882
ASTERISK-5  0x08102363 in pbx_thread (data=0xb68e94d8) at pbx.c:3974
ASTERISK-6 0x081502d0 in dummy_start (data=0xb689b208) at utils.c:861
ASTERISK-7 0x00ae449b in start_thread () from /lib/libpthread.so.0
ASTERISK-8 0x00a3b42e in clone () from /lib/libc.so.6





<<<SECOND CRASH DUMP>>>
#0  0x00e9b402 in __kernel_vsyscall ()
#1  0x00992d80 in raise () from /lib/libc.so.6
#2  0x00994691 in abort () from /lib/libc.so.6
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
#4  0x009d3883 in _int_malloc () from /lib/libc.so.6
ASTERISK-1  0x009d50be in calloc () from /lib/libc.so.6
ASTERISK-2  0x0814e6ab in _ast_calloc (num=1, len=562, file=0x529ed4 "chan_sip.c", lineno=2894, func=0x52b230 "__sip_reliable_xmit")
   at /usr/local/src/asterisk-1.6.0.9/include/asterisk/utils.h:457
ASTERISK-3  0x004c5b16 in __sip_reliable_xmit (p=0xb725b230, seqno=102, resp=0,
   data=0xb71f7f28 "BYE sip:0104f2150dd2@10.3.10.74 SIP/2.0\r\nVia: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-open"..., len=521,
   fatal=0, sipmethod=8) at chan_sip.c:2894
ASTERISK-4  0x004c7398 in send_request (p=0xb725b230, req=0xb71f7d04, reliable=XMIT_RELIABLE, seqno=102) at chan_sip.c:3195
ASTERISK-5  0x004e853f in transmit_request_with_auth (p=0xb725b230, sipmethod=8, seqno=0, reliable=XMIT_RELIABLE, newbranch=1)
   at chan_sip.c:9917
ASTERISK-6 0x004d03ed in sip_hangup (ast=0xb7d42cf0) at chan_sip.c:5010
ASTERISK-7 0x080997f6 in ast_do_masquerade (original=0xb725c980) at channel.c:4159
ASTERISK-8 0x08093bd9 in __ast_read (chan=0xb725c980, dropaudio=0) at channel.c:2526
ASTERISK-9 0x08095906 in ast_read (chan=0xb725c980) at channel.c:2936
ASTERISK-10 0x0811cd65 in bridge_p2p_loop (c0=0xb725c980, c1=0xb7759660, p0=0xb7230c58, p1=0xb77488e8, timeoutms=-1, flags=0,
   fo=0xb71f9d5c, rc=0xb71f9d58, pvt0=0xb725b230, pvt1=0xb774b910) at rtp.c:3668
ASTERISK-11 0x0811ddce in ast_rtp_bridge (c0=0xb725c980, c1=0xb7759660, flags=0, fo=0xb71f9d5c, rc=0xb71f9d58, timeoutms=-1)
   at rtp.c:3925
ASTERISK-12 0x0809c19b in ast_channel_bridge (c0=0xb725c980, c1=0xb7759660, config=0xb71fa9d4, fo=0xb71f9d5c, rc=0xb71f9d58)
   at channel.c:4798
ASTERISK-13 0x080c77af in ast_bridge_call (chan=0xb725c980, peer=0xb7759660, config=0xb71fa9d4) at features.c:2278
ASTERISK-14 0x005f7cda in dial_exec_full (chan=0xb725c980, data=0xb71fcea8, peerflags=0xb71fad30, continue_exec=0x0)
   at app_dial.c:1911
ASTERISK-15 0x005f8a90 in dial_exec (chan=0xb725c980, data=0xb71fcea8) at app_dial.c:1967
ASTERISK-16 0x080f7da2 in pbx_exec (c=0xb725c980, app=0xb7d29eb8, data=0xb71fcea8) at pbx.c:942
ASTERISK-17 0x080fe99b in pbx_extension_helper (c=0xb725c980, con=0x0, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, label=0x0, callerid=0xb7261ca0 "0104f2150dd2", action=E_SPAWN, found=0xb71ff2ec, combined_find_spawn=1)
   at pbx.c:3111
ASTERISK-18 0x08100b28 in ast_spawn_extension (c=0xb725c980, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, callerid=0xb7261ca0 "0104f2150dd2", found=0xb71ff2ec, combined_find_spawn=1) at pbx.c:3614
ASTERISK-19 0x0810117e in __ast_pbx_run (c=0xb725c980, args=0x0) at pbx.c:3701
ASTERISK-20 0x08102363 in pbx_thread (data=0xb725c980) at pbx.c:3974
ASTERISK-21 0x081502d0 in dummy_start (data=0xb72612e0) at utils.c:861
ASTERISK-22 0x00ae449b in start_thread () from /lib/libpthread.so.0
ASTERISK-23 0x00a3b42e in clone () from /lib/libc.so.6

#0  0x00e9b402 in __kernel_vsyscall ()
No symbol table info available.
#1  0x00992d80 in raise () from /lib/libc.so.6
No symbol table info available.
#2  0x00994691 in abort () from /lib/libc.so.6
No symbol table info available.
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
No symbol table info available.
#4  0x009d3883 in _int_malloc () from /lib/libc.so.6
No symbol table info available.
ASTERISK-1  0x009d50be in calloc () from /lib/libc.so.6
No symbol table info available.
ASTERISK-2  0x0814e6ab in _ast_calloc (num=1, len=562, file=0x529ed4 "chan_sip.c", lineno=2894, func=0x52b230 "__sip_reliable_xmit")
   at /usr/local/src/asterisk-1.6.0.9/include/asterisk/utils.h:457
       p = (void *) 0x71
       __PRETTY_FUNCTION__ = "_ast_calloc"
ASTERISK-3  0x004c5b16 in __sip_reliable_xmit (p=0xb725b230, seqno=102, resp=0,
   data=0xb71f7f28 "BYE sip:0104f2150dd2@10.3.10.74 SIP/2.0\r\nVia: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-open"..., len=521,
   fatal=0, sipmethod=8) at chan_sip.c:2894
       pkt = (struct sip_pkt *) 0x0
       siptimer_a = 1000
       xmitres = 0
       __PRETTY_FUNCTION__ = "__sip_reliable_xmit"
ASTERISK-4  0x004c7398 in send_request (p=0xb725b230, req=0xb71f7d04, reliable=XMIT_RELIABLE, seqno=102) at chan_sip.c:3195
       res = 5494584
ASTERISK-5  0x004e853f in transmit_request_with_auth (p=0xb725b230, sipmethod=8, seqno=0, reliable=XMIT_RELIABLE, newbranch=1)
   at chan_sip.c:9917
       resp = {rlPart1 = 0x0, rlPart2 = 0x0, len = 521, headers = 12, method = 8, lines = 0, sdp_start = 0, sdp_end = 0,
 debug = 0 '\0', has_to_tag = 0 '\0', ignore = 0 '\0', header = {
   0xb71f7f28 "BYE sip:0104f2150dd2@10.3.10.74 SIP/2.0\r\nVia: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-open"...,
   0xb71f7f51 "Via: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-openser.nasa.cpwr.corp;user=phone>;tag=as2f11"...,
   0xb71f7f90 "Route: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-openser.nasa.cpwr.corp;user=phone>;tag=as2f11931e\r\nTo: \"0104f2150dd2\" <sip:0104f2150dd2@dtw-openser.nasa.cpw"...,
   0xb71f7fc5 "Max-Forwards: 70\r\nFrom: <sip:18496@dtw-openser.nasa.cpwr.corp;user=phone>;tag=as2f11931e\r\nTo: \"0104f2150dd2\" <sip:0104f2150dd2@dtw-openser.nasa.cpwr.corp>;tag=EAA823F-18B70AC6\r\nCall-ID: 78df5d42-8b1fe"...,
   0xb71f7fd7 "From: <sip:18496@dtw-openser.nasa.cpwr.corp;user=phone>;tag=as2f11931e\r\nTo: \"0104f2150dd2\" <sip:0104f2150dd2@dtw-openser.nasa.cpwr.corp>;tag=EAA823F-18B70AC6\r\nCall-ID: 78df5d42-8b1fe4cd-4d296ebc@10.3."...,
   0xb71f801f "To: \"0104f2150dd2\" <sip:0104f2150dd2@dtw-openser.nasa.cpwr.corp>;tag=EAA823F-18B70AC6\r\nCall-ID: 78df5d42-8b1fe4cd-4d296ebc@10.3.10.74\r\nCSeq: 102 BYE\r\nUser-Agent: Asterisk PBX 1.6.0.9-U1.1\r\nX-Asterisk-"...,
   0xb71f8076 "Call-ID: 78df5d42-8b1fe4cd-4d296ebc@10.3.10.74\r\nCSeq: 102 BYE\r\nUser-Agent: Asterisk PBX 1.6.0.9-U1.1\r\nX-Asterisk-HangupCause: Unknown\r\nX-Asterisk-HangupCauseCode: 0\r\nContent-Length: 0\r\n\r\n",
   0xb71f80a6 "CSeq: 102 BYE\r\nUser-Agent: Asterisk PBX 1.6.0.9-U1.1\r\nX-Asterisk-HangupCause: Unknown\r\nX-Asterisk-HangupCauseCode: 0\r\nContent-Length: 0\r\n\r\n",
   0xb71f80b5 "User-Agent: Asterisk PBX 1.6.0.9-U1.1\r\nX-Asterisk-HangupCause: Unknown\r\nX-Asterisk-HangupCauseCode: 0\r\nContent-Length: 0\r\n\r\n",
   0xb71f80dc "X-Asterisk-HangupCause: Unknown\r\nX-Asterisk-HangupCauseCode: 0\r\nContent-Length: 0\r\n\r\n",
   0xb71f80fd "X-Asterisk-HangupCauseCode: 0\r\nContent-Length: 0\r\n\r\n", 0xb71f811c "Content-Length: 0\r\n\r\n",
   0x0 <repeats 52 times>}, line = {0x0 <repeats 64 times>},
 data = "BYE sip:0104f2150dd2@10.3.10.74 SIP/2.0\r\nVia: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-open"..., socket = {type = 0,
   fd = 0, port = 0, tcptls_session = 0x0}, next = {next = 0x0}}
       __PRETTY_FUNCTION__ = "transmit_request_with_auth"
ASTERISK-6 0x004d03ed in sip_hangup (ast=0xb7d42cf0) at chan_sip.c:5010
       audioqos = 0xb7214010 "ssrc=1223680919;themssrc=1782711874;lp=0;rxjitter=0.000638;rxcount=69;txjitter=0.000000;txcount=0;rlp=0;rtt=0.000000"
       videoqos = 0x52aa04 ""
       textqos = 0x52aa04 ""
       p = (struct sip_pvt *) 0xb725b230
       needcancel = 0
       needdestroy = 0
       oldowner = (struct ast_channel *) 0xb7d42cf0
       __PRETTY_FUNCTION__ = "sip_hangup"
ASTERISK-7 0x080997f6 in ast_do_masquerade (original=0xb725c980) at channel.c:4159
       x = 4
       i = 2
       res = 0
       origstate = 6
       cur = (struct ast_frame *) 0x0
       t = (const struct ast_channel_tech *) 0x53d5c0
       t_pvt = (void *) 0xb725b230
       tmpcid = {cid_dnid = 0x828 <Address 0x828 out of bounds>, cid_num = 0xb71f9288 "È\222\037·\034ç\t\b",
 cid_name = 0x80913f1 "ÉÃU\211å\203ì\b\213E\b\005À", cid_ani = 0xb725cac0 "",
 cid_rdnis = 0x89749d8 "\220û\037·\b¹\222\bT!\030\bÐ\t", cid_pres = -1222667576, cid_ani2 = 134866716,
 cid_ton = -1222260224, cid_tns = 135799124}
       clone = (struct ast_channel *) 0xb7d42cf0
       cdr = (struct ast_cdr *) 0xb7233648
       rformat = 4
       wformat = 4
       newn = "SIP/10.0.12.42-b7d3ebb0\000\001\000\000\000\000\000\000\000ô\237ª\000\000\000\000\000\001\000\000\000X\222\037·Àx®\000h\222\037·Ö|¤\000äQ\227\b\000\000\000\000\210\222\037·\214ù\024\bäQ\227\b"
       orig = "SIP/dtw-openser.nasa.cpwr.corp-b725b230\000\225\223®\000¬t®\000ñb®\0008\222\037·\001\000\000\000¨g\000\000ñb®\000\000\000\000\000\225\223®\000¨g\000"
       masqn = "SIP/10.0.12.42-b7d3ebb0<MASQ>\000\000\000h{p·\024\222\037·\020\000\000\000\002\000HÀ\n\000\f*\000\000\000\000\000\000\000\000¬\000\000\000\020\016\000\200\002\000\000\000|\006\030\b\f\000\000"
       zombn = "\000\000\000\000\000\020\000\000\020\000\000\000\000\000\000\000§\206.J\000\000\000\000\023ûÔI\000\000\000\000\023ûÔI\000\000\000\0001\000Ñ\000\000\000\000\000\020\000p·\231ý¨\000\210\221\037·«S\235\000\020\000p·\017\000\000\000ô\237ª\000\017\000\000"
       __PRETTY_FUNCTION__ = "ast_do_masquerade"
ASTERISK-8 0x08093bd9 in __ast_read (chan=0xb725c980, dropaudio=0) at channel.c:2526
       f = (struct ast_frame *) 0x0
       blah = 6
       prestate = 6
       count = 1
       __PRETTY_FUNCTION__ = "__ast_read"
ASTERISK-9 0x08095906 in ast_read (chan=0xb725c980) at channel.c:2936
No locals.
ASTERISK-10 0x0811cd65 in bridge_p2p_loop (c0=0xb725c980, c1=0xb7759660, p0=0xb7230c58, p1=0xb77488e8, timeoutms=-1, flags=0,
   fo=0xb71f9d5c, rc=0xb71f9d58, pvt0=0xb725b230, pvt1=0xb774b910) at rtp.c:3668
       fr = (struct ast_frame *) 0x81b7100
       who = (struct ast_channel *) 0xb725c980
       other = (struct ast_channel *) 0xb725c980
       cs = {0xb7759660, 0xb725c980, 0xb725c980}
       p0_iod = {0x0, 0x0}
       p1_iod = {0x0, 0x0}
       p0_callback = 0
       p1_callback = 0
       res = AST_BRIDGE_FAILED
       __PRETTY_FUNCTION__ = "bridge_p2p_loop"
ASTERISK-11 0x0811ddce in ast_rtp_bridge (c0=0xb725c980, c1=0xb7759660, flags=0, fo=0xb71f9d5c, rc=0xb71f9d58, timeoutms=-1)
   at rtp.c:3925
       fmt0 = {bits = 4, name = 0x818fd21 "ulaw", samplespersecond = 8000, desc = 0x818fd26 "G.711 u-law", fr_len = 80,
 min_ms = 10, max_ms = 150, inc_ms = 10, def_ms = 20, flags = 0, cur_ms = 20}
       fmt1 = {bits = 4, name = 0x818fd21 "ulaw", samplespersecond = 8000, desc = 0x818fd26 "G.711 u-law", fr_len = 80,
 min_ms = 10, max_ms = 150, inc_ms = 10, def_ms = 20, flags = 0, cur_ms = 20}
       p0 = (struct ast_rtp *) 0xb7230c58
       p1 = (struct ast_rtp *) 0xb77488e8
       vp0 = (struct ast_rtp *) 0x0
       vp1 = (struct ast_rtp *) 0x0
       tp0 = (struct ast_rtp *) 0x0
       tp1 = (struct ast_rtp *) 0x0
       pr0 = (struct ast_rtp_protocol *) 0x53e550
       pr1 = (struct ast_rtp_protocol *) 0x53e550
       audio_p0_res = AST_RTP_TRY_PARTIAL
       video_p0_res = AST_RTP_GET_FAILED
       text_p0_res = AST_RTP_GET_FAILED
       audio_p1_res = AST_RTP_TRY_PARTIAL
       video_p1_res = AST_RTP_GET_FAILED
       text_p1_res = AST_RTP_GET_FAILED
       res = AST_BRIDGE_FAILED
       codec0 = 4364
       codec1 = 4
       pvt0 = (void *) 0xb725b230
       pvt1 = (void *) 0xb774b910
       __PRETTY_FUNCTION__ = "ast_rtp_bridge"
ASTERISK-12 0x0809c19b in ast_channel_bridge (c0=0xb725c980, c1=0xb7759660, config=0xb71fa9d4, fo=0xb71f9d5c, rc=0xb71f9d58)
   at channel.c:4798
       now = {tv_sec = 0, tv_usec = 0}
       to = -1
       who = (struct ast_channel *) 0x0
       res = AST_BRIDGE_COMPLETE
       nativefailed = 0
       firstpass = 1
       o0nativeformats = 4
       o1nativeformats = 4
       time_left_ms = 0
       caller_warning = 0 '\0'
       callee_warning = 0 '\0'
       __PRETTY_FUNCTION__ = "ast_channel_bridge"
ASTERISK-13 0x080c77af in ast_bridge_call (chan=0xb725c980, peer=0xb7759660, config=0xb71fa9d4) at features.c:2278
       other = (struct ast_channel *) 0x0
       f = (struct ast_frame *) 0x0
       who = (struct ast_channel *) 0xb71f9dc8
       chan_featurecode = '\0' <repeats 11 times>
       peer_featurecode = '\0' <repeats 11 times>
       orig_channame = "SIP/dtw-openser.nasa.cpwr.corp-b725b230\0008\235\037·\214ù\024\bäQ\227\b(\b\000\0008\235\037·ñ\023\t\bÀÊ%·ØI\227\bx\235\037·Àá\t\b"
       orig_peername = "SIP/dtw-openser.nasa.cpwr.corp-b774b910\000Ø\234\037·Ö|¤\000¬t®\000\220û\037·\b\235\037·\001\000\000\000äQ\227\bô\237ª\000\000\000\000\000\001\000\000"
       res = 135799124
       diff = 1859
       hasfeatures = 0
       hadfeatures = 0
       autoloopflag = -1222260224
       aoh = (struct ast_option_header *) 0x5fe120
       backup_config = {features_caller = {flags = 0}, features_callee = {flags = 0}, start_time = {tv_sec = 0,
   tv_usec = 0}, nexteventts = {tv_sec = 0, tv_usec = 0}, feature_timer = 0, timelimit = 0, play_warning = 0,
 warning_freq = 0, warning_sound = 0x0, end_sound = 0x0, start_sound = 0x0, firstpass = 0, flags = 0,
 end_bridge_callback = 0, end_bridge_callback_data = 0x0, end_bridge_callback_data_fixup = 0}
       bridge_cdr = (struct ast_cdr *) 0xb77a6c28
       orig_peer_cdr = (struct ast_cdr *) 0xb7708b00
       chan_cdr = (struct ast_cdr *) 0xb7233648
       peer_cdr = (struct ast_cdr *) 0xb7708b00
       new_chan_cdr = (struct ast_cdr *) 0x0
       new_peer_cdr = (struct ast_cdr *) 0x0
       __PRETTY_FUNCTION__ = "ast_bridge_call"
ASTERISK-14 0x005f7cda in dial_exec_full (chan=0xb725c980, data=0xb71fcea8, peerflags=0xb71fad30, continue_exec=0x0)
   at app_dial.c:1911
       number = 0xb77482f9 "18496@dtw-openser.nasa.cpwr.corp"
       res = 0
       rest = 0x0
       cur = 0x0
       outgoing = (struct chanlist *) 0x0
       peer = (struct ast_channel *) 0xb7759660
       to = -1
       num = {chan = 0xb725c980, busy = 0, congestion = 0, nochan = 0}
       cause = 0
       numsubst = "18496@dtw-openser.nasa.cpwr.corp\000\000\000\000¨g\000\000@\000 ·t¬1244175387èÍ\037·\004\000\000\000à\217ª\000\v\000\000\000\000\000\000\000\017\000\000\000\b['·\000\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000ôó\2343ñ(\235\000\000\000\000\000ô\237ª\000\000\000\000\000\000\000\000\000\235\006\235\000,«\037·ä¬\037·+­\037·\214¬\037·ô\237ª\000ôó\234\000ä¬\037·\030¬\037·ô\237ª\000\000\000\000\000\000\000\000\000\235\006\235\000\\«\037·é\206\215\bE\207\215\b¼¬\037·ô\237ª\000\\«\037·"...
       cidname = '\0' <repeats 79 times>
       config = {features_caller = {flags = 0}, features_callee = {flags = 0}, start_time = {tv_sec = 1244563111,
   tv_usec = 442693}, nexteventts = {tv_sec = 0, tv_usec = 0}, feature_timer = 0, timelimit = 0, play_warning = 0,
 warning_freq = 0, warning_sound = 0x0, end_sound = 0x0, start_sound = 0x0, firstpass = 0, flags = 0,
 end_bridge_callback = 0x5f37cb <end_bridge_callback>, end_bridge_callback_data = 0xb725c980,
 end_bridge_callback_data_fixup = 0x5f3923 <end_bridge_callback_data_fixup>}
       calldurationlimit = 0
       dtmfcalled = 0x0
       dtmfcalling = 0x0
       pa = {sentringing = 0, privdb_val = 0, privcid = '\0' <repeats 255 times>, privintro = '\0' <repeats 1023 times>,
 status = "ANSWER\000R\000GS", '\0' <repeats 244 times>}
       sentringing = 0
       moh = 0
       outbound_group = 0x0
       result = 0
       parse = 0xb71f9e70 "SIP"
       opermode = 0
       args = {argc = 1, argv = 0xb71fa3b0, peers = 0xb71f9e70 "SIP", timeout = 0x0, options = 0x0, url = 0x0}
       opts = {flags = 0}
       opt_args = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20 <Address 0x20 out of bounds>, 0x0}
       datastore = (struct ast_datastore *) 0xb7707c78
       fulldial = 0
       num_dialed = 1
       __PRETTY_FUNCTION__ = "dial_exec_full"
ASTERISK-15 0x005f8a90 in dial_exec (chan=0xb725c980, data=0xb71fcea8) at app_dial.c:1967
       peerflags = {flags = 4294967296}
ASTERISK-16 0x080f7da2 in pbx_exec (c=0xb725c980, app=0xb7d29eb8, data=0xb71fcea8) at pbx.c:942
       res = 0
       u = (struct ast_module_user *) 0xb7275b00
       saved_c_appl = 0x0
       saved_c_data = 0x0
ASTERISK-17 0x080fe99b in pbx_extension_helper (c=0xb725c980, con=0x0, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, label=0x0, callerid=0xb7261ca0 "0104f2150dd2", action=E_SPAWN, found=0xb71ff2ec, combined_find_spawn=1)
   at pbx.c:3111
       e = (struct ast_exten *) 0x8772c90
       app = (struct ast_app *) 0xb7d29eb8
       res = -1222640752
       q = {incstack = {0x0 <repeats 128 times>}, stacklen = 0, status = 5, swo = 0x0, data = 0x0,
 foundcontext = 0xb725cbc8 "b2b_agent_dialer"}
       passdata = "SIP/18496@dtw-openser.nasa.cpwr.corp\000bab0)\000bers\000al/bab6@queuemembers\000ne?reason=%20sip%3bcause%3d480%3btext%3d%20%22Temporarily%20Unavailable%22>;index=1", '\0' <repeats 1048 times>, "Sõ\234", '\0' <repeats 13 times>, "~\r\234\000\024Ù\037·\000\000\000\000\035Û\037·ô\237ª\000ºEe\000\000"...
       matching_action = 0
       __PRETTY_FUNCTION__ = "pbx_extension_helper"
ASTERISK-18 0x08100b28 in ast_spawn_extension (c=0xb725c980, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, callerid=0xb7261ca0 "0104f2150dd2", found=0xb71ff2ec, combined_find_spawn=1) at pbx.c:3614
No locals.
ASTERISK-19 0x0810117e in __ast_pbx_run (c=0xb725c980, args=0x0) at pbx.c:3701
       dst_exten = "ô\237ª\000@±ª\000¸\v\002\000\bò\037·½3\235", '\0' <repeats 29 times>, "\200ån\b°c\213\bóñ\037·p³ª\000\000\000\000\000\000\000\000\000p±ª\000\000\000\000\0001\b\000\000\000\000\000\000\002\000\000\000\005", '\0' <repeats 11 times>, "\220û\037·\bó\037·Ð\233Ã\000ðñ\037·pò\037·\200\000\000\000\030ó\037·ðñ\037·p±ª\000ÐI\227\bX\000\000\000Àý\037·\000\000\000\000\000\000\000\000q2\235\000ôò\037·\000\000\000\000øò\037·ô\237ª\000@±ª\000¸\v\002\000ØI\227\bÜQ\235\000ØI\227\b\000\000\000\000,\b\000\000@±ª\000Hô\236\b(\b\000\000"...
       pos = 0
       digit = 0
       found = 1
       res = 0
       autoloopflag = 0
       error = 0
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
ASTERISK-20 0x08102363 in pbx_thread (data=0xb725c980) at pbx.c:3974
       c = (struct ast_channel *) 0xb725c980
ASTERISK-21 0x081502d0 in dummy_start (data=0xb72612e0) at utils.c:861
       __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {11481076, 0, -1222640752, -1222642760, -1306630584,
       219452614}, __mask_was_saved = 0}}, __pad = {0xb71ff470, 0x0, 0x89bf8c8, 0x88aab50}}
       __cancel_routine = (void (*)(void *)) 0x80720a1 <ast_unregister_thread>
       __cancel_arg = (void *) 0xb71ffb90
       not_first_call = 0
       ret = (void *) 0xb71ff3b4
       a = {start_routine = 0x8102344 <pbx_thread>, data = 0xb725c980,
 name = 0xb7227578 "pbx_thread", ' ' <repeats 11 times>, "started at [ 3995] pbx.c ast_pbx_start()"}
       lock_info = (struct thr_lock_info *) 0x89749d8
       mutex_attr = {__size = "\001\000\000", __align = 1}
ASTERISK-22 0x00ae449b in start_thread () from /lib/libpthread.so.0
No symbol table info available.
ASTERISK-23 0x00a3b42e in clone () from /lib/libc.so.6
No symbol table info available.



(gdb) thread apply all bt

Thread 1 (process 26536):
#0  0x00e9b402 in __kernel_vsyscall ()
#1  0x00992d80 in raise () from /lib/libc.so.6
#2  0x00994691 in abort () from /lib/libc.so.6
#3  0x009cb24b in __libc_message () from /lib/libc.so.6
#4  0x009d3883 in _int_malloc () from /lib/libc.so.6
ASTERISK-1  0x009d50be in calloc () from /lib/libc.so.6
ASTERISK-2  0x0814e6ab in _ast_calloc (num=1, len=562, file=0x529ed4 "chan_sip.c", lineno=2894, func=0x52b230 "__sip_reliable_xmit")
   at /usr/local/src/asterisk-1.6.0.9/include/asterisk/utils.h:457
ASTERISK-3  0x004c5b16 in __sip_reliable_xmit (p=0xb725b230, seqno=102, resp=0,
   data=0xb71f7f28 "BYE sip:0104f2150dd2@10.3.10.74 SIP/2.0\r\nVia: SIP/2.0/UDP 10.0.12.62:5060;branch=z9hG4bK715722af;rport\r\nRoute: <sip:10.0.12.52;lr=on;ftag=EAA823F-18B70AC6>\r\nMax-Forwards: 70\r\nFrom: <sip:18496@dtw-open"..., len=521,
   fatal=0, sipmethod=8) at chan_sip.c:2894
ASTERISK-4  0x004c7398 in send_request (p=0xb725b230, req=0xb71f7d04, reliable=XMIT_RELIABLE, seqno=102) at chan_sip.c:3195
ASTERISK-5  0x004e853f in transmit_request_with_auth (p=0xb725b230, sipmethod=8, seqno=0, reliable=XMIT_RELIABLE, newbranch=1)
   at chan_sip.c:9917
ASTERISK-6 0x004d03ed in sip_hangup (ast=0xb7d42cf0) at chan_sip.c:5010
ASTERISK-7 0x080997f6 in ast_do_masquerade (original=0xb725c980) at channel.c:4159
ASTERISK-8 0x08093bd9 in __ast_read (chan=0xb725c980, dropaudio=0) at channel.c:2526
ASTERISK-9 0x08095906 in ast_read (chan=0xb725c980) at channel.c:2936
ASTERISK-10 0x0811cd65 in bridge_p2p_loop (c0=0xb725c980, c1=0xb7759660, p0=0xb7230c58, p1=0xb77488e8, timeoutms=-1, flags=0,
   fo=0xb71f9d5c, rc=0xb71f9d58, pvt0=0xb725b230, pvt1=0xb774b910) at rtp.c:3668
ASTERISK-11 0x0811ddce in ast_rtp_bridge (c0=0xb725c980, c1=0xb7759660, flags=0, fo=0xb71f9d5c, rc=0xb71f9d58, timeoutms=-1)
   at rtp.c:3925
ASTERISK-12 0x0809c19b in ast_channel_bridge (c0=0xb725c980, c1=0xb7759660, config=0xb71fa9d4, fo=0xb71f9d5c, rc=0xb71f9d58)
   at channel.c:4798
ASTERISK-13 0x080c77af in ast_bridge_call (chan=0xb725c980, peer=0xb7759660, config=0xb71fa9d4) at features.c:2278
ASTERISK-14 0x005f7cda in dial_exec_full (chan=0xb725c980, data=0xb71fcea8, peerflags=0xb71fad30, continue_exec=0x0)
   at app_dial.c:1911
ASTERISK-15 0x005f8a90 in dial_exec (chan=0xb725c980, data=0xb71fcea8) at app_dial.c:1967
ASTERISK-16 0x080f7da2 in pbx_exec (c=0xb725c980, app=0xb7d29eb8, data=0xb71fcea8) at pbx.c:942
ASTERISK-17 0x080fe99b in pbx_extension_helper (c=0xb725c980, con=0x0, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, label=0x0, callerid=0xb7261ca0 "0104f2150dd2", action=E_SPAWN, found=0xb71ff2ec, combined_find_spawn=1)
   at pbx.c:3111
ASTERISK-18 0x08100b28 in ast_spawn_extension (c=0xb725c980, context=0xb725cbc8 "b2b_agent_dialer", exten=0xb725cc18 "s",
   priority=3, callerid=0xb7261ca0 "0104f2150dd2", found=0xb71ff2ec, combined_find_spawn=1) at pbx.c:3614
ASTERISK-19 0x0810117e in __ast_pbx_run (c=0xb725c980, args=0x0) at pbx.c:3701
ASTERISK-20 0x08102363 in pbx_thread (data=0xb725c980) at pbx.c:3974
ASTERISK-21 0x081502d0 in dummy_start (data=0xb72612e0) at utils.c:861
ASTERISK-22 0x00ae449b in start_thread () from /lib/libpthread.so.0
ASTERISK-23 0x00a3b42e in clone () from /lib/libc.so.6


Comments:By: Leif Madsen (lmadsen) 2009-06-16 15:44:00

As 1.6.0.9 is actually starting to get a bit old now (we're going to be releasing 1.6.0.11 soon) I'd be curious if you're getting the same crashing issues on the latest 1.6.0 branch?

svn co http://svn.asterisk.org/svn/asterisk/branches/1.6.0

If you could install that and verify if you're getting the same crashing issues, that would be tops!
By: Kevin Scott Adams (nivek) 2009-06-20 07:21:56

It is a production system.  I will see what I can pull off!!
By: Kevin Scott Adams (nivek) 2009-06-30 11:30:58

I saw the Change Log for 1.6.0.11-rc1 (God I miss ViewVC).
It does seem to have a "few" updates that could relate to our issue.
We our currently in the end of FY10Q1 so the change control window is in "blackout" mode.

I will see if I can get it in on the 4th of July weekend.

Thanks for you help.
By: Leif Madsen (lmadsen) 2009-07-13 10:35:33

Any luck? Thanks!
By: Russell Bryant (russell) 2009-09-10 22:32:07

Feel free to reopen if this is still a problem with the latest version.  Thanks!