ASTERISK-04847: crash with iax2 debug

[Home]

Summary: ASTERISK-04847: crash with iax2 debug

Reporter: Bobby Krupczak (rdk) Labels:

Date Opened: 2005-08-16 18:10:22 Date Closed: 2011-06-07 14:00:48

Priority: Critical Regression? No

Status: Closed/Complete Components: Core/General

Versions: Frequency of
Occurrence

Related
Issues:

Environment: Attachments: ( 0) iax2.debug.tcpdump

Description: I run asterisk in background

I connect to console via -r

I enable iax2 debug

Dial a phone number and asterisk crashes/exits

I am connected to VoicePulse via iax2 and am dialing a different VoicePulse
user.

****** ADDITIONAL INFORMATION ******

Linux, x86, FC4 machine with Digium TDM card

I am trying to debug my configuration and am not getting calls
to go through from VoicePulse to VoicePulse. Dont know why.

Comments: By: Mark Spencer (markster) 2005-08-16 18:49:17

Making a core requires running with "-g" option.
By: Bobby Krupczak (rdk) 2005-08-16 18:53:04

I will re-run and see if a core is left.

[Thanks for coming to the Atlanta AUUG meeting earlier this year. I am a
friend of jeff and briefly met you after the meeting.]
By: Bobby Krupczak (rdk) 2005-08-16 19:01:33

I re-started asterisk with -g
No core is produced (anywhere that I can find) but asterisk
and the console both exit abrubtly. I am stumped.
By: Bobby Krupczak (rdk) 2005-08-16 21:15:00

Strike that last note; I did indeed find a core file; I uploaded it.
Thanks
By: Mark Spencer (markster) 2005-08-16 21:19:09

Don't attach the core, attach a backtrace as per the bug guidelines. Thanks
By: Bobby Krupczak (rdk) 2005-08-16 21:47:38

(gdb) bt full
#0 0x0047372f in vfprintf () from /lib/libc.so.6
No symbol table info available.
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
No symbol table info available.
#2 0x00478be1 in snprintf () from /lib/libc.so.6
No symbol table info available.
#3 0x00fda318 in iax_showframe (f=0x0, fhi=0xb6bfa9a4, rx=1, sin=0xb6bfd3a8,
datalen=0) at iax2-parser.c:403
frames = {0xfe0675 "(0?)", 0xfe07d0 "DTMF ", 0xfe07d8 "VOICE ",
0xfe07e0 "VIDEO ", 0xfdfaa1 "CONTROL", 0xfe07e8 "NULL ",
0xfe07f0 "IAX ", 0xfe07f8 "TEXT ", 0xfe0800 "IMAGE ",
0xfe0808 "HTML ", 0xfe0810 "CNG "}
iaxs = {0xfe0675 "(0?)", 0xfe06b3 "NEW ", 0xfe06bb "PING ",
0xfe06c3 "PONG ", 0xfe06cb "ACK ", 0xfe067a "HANGUP ",
0xfe06d3 "REJECT ", 0xfe06db "ACCEPT ", 0xfe06e3 "AUTHREQ",
0xfe06eb "AUTHREP", 0xfe06f3 "INVAL ", 0xfe06fb "LAGRQ ",
0xfe0703 "LAGRP ", 0xfe070b "REGREQ ", 0xfe0713 "REGAUTH",
0xfe071b "REGACK ", 0xfe0723 "REGREJ ", 0xfe072b "REGREL ",
0xfe0733 "VNAK ", 0xfe073b "DPREQ ", 0xfe0743 "DPREP ",
0xfe074b "DIAL ", 0xfe0753 "TXREQ ", 0xfe075b "TXCNT ",
0xfe0763 "TXACC ", 0xfe076b "TXREADY", 0xfe0773 "TXREL ",
0xfe077b "TXREJ ", 0xfe0783 "QUELCH ", 0xfe078b "UNQULCH", 0xfe0793 "POKE",
0xfe0798 "PAGE", 0xfe079d "MWI", 0xfe07a1 "UNSUPPORTED",
0xfe07ad "TRANSFER", 0xfe07b6 "PROVISION", 0xfe07c0 "FWDOWNLD",
0xfe07c9 "FWDATA"}
cmds = {0xfe0675 "(0?)", 0xfe067a "HANGUP ", 0xfe0682 "RING ",
0xfe068a "RINGING", 0xfe0692 "ANSWER ", 0xfe069a "BUSY ",
0xfe06a2 "TKOFFHK ", 0xfe06ab "OFFHOOK"}
fh = (struct ast_iax2_full_hdr *) 0xb6bfa9a4
retries = " No\000\"±ý\000\001\000\000\000¸Ñ¿¶\b§¿¶"
class2 = "\200ÈU\000¸Ñ¿¶è¦¿¶\222§I\000\200ÈU"
subclass2 = "(255?)\000\000\000\000\000\000\bÂ\211\bô¯U"
class = 0xfdfaa1 "CONTROL"
subclass = Variable "subclass" is not available.

(gdb) where
#0 0x0047372f in vfprintf () from /lib/libc.so.6
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
#2 0x00478be1 in snprintf () from /lib/libc.so.6
#3 0x00fda318 in iax_showframe (f=0x0, fhi=0xb6bfa9a4, rx=1, sin=0xb6bfd3a8,
datalen=0) at iax2-parser.c:403
#4 0x00fcfcd4 in socket_read (id=0x8862260, fd=11, events=1, cbdata=0x0)
at chan_iax2.c:5157
ASTERISK-1 0x08052bd4 in ast_io_wait (ioc=0x8898058, howlong=0) at io.c:267
ASTERISK-2 0x00fc2923 in network_thread (ignore=0x0) at chan_iax2.c:6388
ASTERISK-3 0x005a7b80 in start_thread () from /lib/libpthread.so.0
ASTERISK-4 0x004ffdee in clone () from /lib/libc.so.6
By: Clod Patry (junky) 2005-08-17 01:02:32

CVS-Nv1-0-9-08/10/05 is not CVS HEAD btw.

not sure why it's like that:
#ifdef DEBUG_SUPPORT
if (iaxdebug)
iax_showframe(NULL, fh, 1, &sin, res - sizeof(struct ast_iax2_full_hdr));
#endif

it's explicitly setting the ast_frame to NULL, maybe kram could answers this.
By: Bobby Krupczak (rdk) 2005-08-17 06:23:19

Sorry, I'm new to this and did not know what CVS head was.

I went with this rev because I had problems building asterisk with the
version of zaptel drivers that came with my card.

Is there a combo of zaptel driver version and asterisk version that
I should use instead of the one I've built? I'd be happy to switch. Thanks.
By: Michael Jerris (mikej) 2005-08-17 08:31:45

can you clarify what version you are using, you say head, then you say 1.0.9.
By: Bobby Krupczak (rdk) 2005-08-17 08:44:43

CVS-Nv1-0-9-08/10/05 is the version asterisk shows when it starts up
By: Michael Jerris (mikej) 2005-08-17 08:49:31

can you test this with cvs head and see if it is an issue there as well.
By: Bobby Krupczak (rdk) 2005-08-17 08:57:10

I went and downloaded 1.0.9 from asterisk.org and built and installed. This src tar was not pulled from cvs.

Does this help?

I get the same core with iax2 debug enabled. Here is back trace:
(gdb) bt full
#0 0x0047372f in vfprintf () from /lib/libc.so.6
No symbol table info available.
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
No symbol table info available.
#2 0x00478be1 in snprintf () from /lib/libc.so.6
No symbol table info available.
#3 0x01018318 in iax_showframe (f=0x0, fhi=0xb6d029a4, rx=1, sin=0xb6d053a8,
datalen=0) at iax2-parser.c:403
frames = {0x101e675 "(0?)", 0x101e7d0 "DTMF ", 0x101e7d8 "VOICE ",
0x101e7e0 "VIDEO ", 0x101daa1 "CONTROL", 0x101e7e8 "NULL ",
0x101e7f0 "IAX ", 0x101e7f8 "TEXT ", 0x101e800 "IMAGE ",
0x101e808 "HTML ", 0x101e810 "CNG "}
iaxs = {0x101e675 "(0?)", 0x101e6b3 "NEW ", 0x101e6bb "PING ",
0x101e6c3 "PONG ", 0x101e6cb "ACK ", 0x101e67a "HANGUP ",
0x101e6d3 "REJECT ", 0x101e6db "ACCEPT ", 0x101e6e3 "AUTHREQ",
0x101e6eb "AUTHREP", 0x101e6f3 "INVAL ", 0x101e6fb "LAGRQ ",
0x101e703 "LAGRP ", 0x101e70b "REGREQ ", 0x101e713 "REGAUTH",
0x101e71b "REGACK ", 0x101e723 "REGREJ ", 0x101e72b "REGREL ",
0x101e733 "VNAK ", 0x101e73b "DPREQ ", 0x101e743 "DPREP ",
0x101e74b "DIAL ", 0x101e753 "TXREQ ", 0x101e75b "TXCNT ",
0x101e763 "TXACC ", 0x101e76b "TXREADY", 0x101e773 "TXREL ",
0x101e77b "TXREJ ", 0x101e783 "QUELCH ", 0x101e78b "UNQULCH",
0x101e793 "POKE", 0x101e798 "PAGE", 0x101e79d "MWI",
0x101e7a1 "UNSUPPORTED", 0x101e7ad "TRANSFER", 0x101e7b6 "PROVISION",
0x101e7c0 "FWDOWNLD", 0x101e7c9 "FWDATA"}
cmds = {0x101e675 "(0?)", 0x101e67a "HANGUP ", 0x101e682 "RING ",
0x101e68a "RINGING", 0x101e692 "ANSWER ", 0x101e69a "BUSY ",
0x101e6a2 "TKOFFHK ", 0x101e6ab "OFFHOOK"}
fh = (struct ast_iax2_full_hdr *) 0xb6d029a4
retries = " No\000\"\221\001\001\001\000\000\000¸QÐ¶\b'Ð¶"
class2 = "\200ÈU\000¸QÐ¶è&Ð¶\222§I\000\200ÈU"
subclass2 = "(255?)\000\000\000\000\000\000 ÷\234\tô¯U"
class = 0x101daa1 "CONTROL"
subclass = Variable "subclass" is not available.

(gdb) where
#0 0x0047372f in vfprintf () from /lib/libc.so.6
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
#2 0x00478be1 in snprintf () from /lib/libc.so.6
#3 0x01018318 in iax_showframe (f=0x0, fhi=0xb6d029a4, rx=1, sin=0xb6d053a8,
datalen=0) at iax2-parser.c:403
#4 0x0100dcd4 in socket_read (id=0x99cf260, fd=12, events=1, cbdata=0x0)
at chan_iax2.c:5157
ASTERISK-1 0x08052bd4 in ast_io_wait (ioc=0x9a05090, howlong=0) at io.c:267
ASTERISK-2 0x01000923 in network_thread (ignore=0x0) at chan_iax2.c:6388
ASTERISK-3 0x005a7b80 in start_thread () from /lib/libpthread.so.0
ASTERISK-4 0x004ffdee in clone () from /lib/libc.so.6
By: Michael Jerris (mikej) 2005-08-17 09:05:31

well, that is the same as before, as they are both 1.0.9. What I would like is for you to download cvs head from cvs, compile and re-test. If you need help with this, pop in to #asterisk-bugs on irc and somone can give you a hand.
By: Bobby Krupczak (rdk) 2005-08-17 10:16:28

OK, checked out CVS head and got CVS-NHEAD-08/17/05-11:01:30

Now, when I start up asterisk, it cores immediately and never lets
me get to where I can test iax2.

Do think think there is a problem with my configs?

Help!

(gdb) where
#0 0x005aa5bb in pthread_cancel () from /lib/libpthread.so.0
#1 0x00175ca7 in unload_module () at chan_sip.c:11660
#2 0x0805ad79 in ast_unload_resource (resource_name=0x884e397 "chan_sip.so",
force=0) at loader.c:129
#3 0x0805b7c7 in __load_resource (resource_name=0x884e397 "chan_sip.so", cfg=Variable "cfg" is not available.
)
at loader.c:404
#4 0x0805bbe4 in load_modules (preload_only=0) at loader.c:542
ASTERISK-1 0x080b1df9 in main (argc=3, argv=0xbf8c8054) at asterisk.c:2110
(gdb) bt full
#0 0x005aa5bb in pthread_cancel () from /lib/libpthread.so.0
No symbol table info available.
#1 0x00175ca7 in unload_module () at chan_sip.c:11660
newcount = Variable "newcount" is not available.
By: Michael Jerris (mikej) 2005-08-17 10:21:16

remove all your so modules before going from stable to head.
By: Bobby Krupczak (rdk) 2005-08-17 10:24:13

I checked the timestamps and they all were updated when I did a make install
with the head version.

Is there something else I missed?
By: Bobby Krupczak (rdk) 2005-08-17 10:27:32

When I checked out asterisk from cvs, should I also have checked out
the zaptel drivers?
By: Michael Jerris (mikej) 2005-08-17 10:32:26

yes, as well as libpri.
By: Bobby Krupczak (rdk) 2005-08-17 14:20:54

I downloaded CVS HEAD and re-built/installed zaptel, libpri, and asterisk
from scratch.

I am still getting the same core dump now during startup/loading. Please help! I'm ready to throw in the towel.

Now, one thing I did was to build with my install root being /asterisk.

(gdb) where
#0 0x005aa5bb in pthread_cancel () from /lib/libpthread.so.0
#1 0x00133ca7 in unload_module () at chan_sip.c:11660
#2 0x0805ad79 in ast_unload_resource (resource_name=0x97e63b7 "chan_sip.so",
force=0) at loader.c:129
#3 0x0805b7c7 in __load_resource (resource_name=0x97e63b7 "chan_sip.so", cfg=Variable "cfg" is not available.
)
at loader.c:404
#4 0x0805bbe4 in load_modules (preload_only=0) at loader.c:542
ASTERISK-1 0x080b1ed9 in main (argc=3, argv=0xbf930674) at asterisk.c:2110

(gdb) bt full
#0 0x005aa5bb in pthread_cancel () from /lib/libpthread.so.0
No symbol table info available.
#1 0x00133ca7 in unload_module () at chan_sip.c:11660
newcount = Variable "newcount" is not available.
By: Bobby Krupczak (rdk) 2005-08-17 17:58:37

I went and un-installed everything I could find and wiped out
configs and re-built everything using 1.0.9 (zaptel, libpri, asterisk)
and the problem has now disappeared.

One big change from all my previous builds -- I took *all* the
default install locations and did not edit a single Makefile.

I'm wondering if the problem is related to trying to build/install
asterisk et al in /asterisk rather than all the usual locations.
By: Clod Patry (junky) 2005-08-17 18:27:48

so no more crash with CVS-HEAD?
By: Bobby Krupczak (rdk) 2005-08-22 16:40:14

Still crashes on iax2 debug. I am therefore unable to debug connection problems
with voicepulse.

I have tried the CVS-Head, 1.0.9, building them in the default locations
and building them in /asterisk.

Its difficult to keep switching versions. My users are not happy.

Bobby

(gdb) where
#0 0x0047372f in vfprintf () from /lib/libc.so.6
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
#2 0x00478be1 in snprintf () from /lib/libc.so.6
#3 0x00fd64f8 in iax_showframe (f=0x0, fhi=0xb73409a4, rx=1, sin=0xb73433a8,
datalen=0) at iax2-parser.c:403
#4 0x00fcbeb4 in socket_read (id=0x8bb7ad8, fd=10, events=1, cbdata=0x0)
at chan_iax2.c:5157
ASTERISK-1 0x08052bd4 in ast_io_wait (ioc=0x8bced08, howlong=0) at io.c:267
ASTERISK-2 0x00fbe953 in network_thread (ignore=0x0) at chan_iax2.c:6388
ASTERISK-3 0x005a7b80 in start_thread () from /lib/libpthread.so.0
ASTERISK-4 0x004ffdee in clone () from /lib/libc.so.6

(gdb) bt full
#0 0x0047372f in vfprintf () from /lib/libc.so.6
No symbol table info available.
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
No symbol table info available.
#2 0x00478be1 in snprintf () from /lib/libc.so.6
No symbol table info available.
#3 0x00fd64f8 in iax_showframe (f=0x0, fhi=0xb73409a4, rx=1, sin=0xb73433a8,
datalen=0) at iax2-parser.c:403
frames = {0xfdc8f5 "(0?)", 0xfdca50 "DTMF ", 0xfdca58 "VOICE ",
0xfdca60 "VIDEO ", 0xfdbcfd "CONTROL", 0xfdca68 "NULL ",
0xfdca70 "IAX ", 0xfdca78 "TEXT ", 0xfdca80 "IMAGE ",
0xfdca88 "HTML ", 0xfdca90 "CNG "}
iaxs = {0xfdc8f5 "(0?)", 0xfdc933 "NEW ", 0xfdc93b "PING ",
0xfdc943 "PONG ", 0xfdc94b "ACK ", 0xfdc8fa "HANGUP ",
0xfdc953 "REJECT ", 0xfdc95b "ACCEPT ", 0xfdc963 "AUTHREQ",
0xfdc96b "AUTHREP", 0xfdc973 "INVAL ", 0xfdc97b "LAGRQ ",
0xfdc983 "LAGRP ", 0xfdc98b "REGREQ ", 0xfdc993 "REGAUTH",
0xfdc99b "REGACK ", 0xfdc9a3 "REGREJ ", 0xfdc9ab "REGREL ",
0xfdc9b3 "VNAK ", 0xfdc9bb "DPREQ ", 0xfdc9c3 "DPREP ",
0xfdc9cb "DIAL ", 0xfdc9d3 "TXREQ ", 0xfdc9db "TXCNT ",
0xfdc9e3 "TXACC ", 0xfdc9eb "TXREADY", 0xfdc9f3 "TXREL ",
0xfdc9fb "TXREJ ", 0xfdca03 "QUELCH ", 0xfdca0b "UNQULCH", 0xfdca13 "POKE",
0xfdca18 "PAGE", 0xfdca1d "MWI", 0xfdca21 "UNSUPPORTED",
0xfdca2d "TRANSFER", 0xfdca36 "PROVISION", 0xfdca40 "FWDOWNLD",
0xfdca49 "FWDATA"}
cmds = {0xfdc8f5 "(0?)", 0xfdc8fa "HANGUP ", 0xfdc902 "RING ",
0xfdc90a "RINGING", 0xfdc912 "ANSWER ", 0xfdc91a "BUSY ",
0xfdc922 "TKOFFHK ", 0xfdc92b "OFFHOOK"}
fh = (struct ast_iax2_full_hdr *) 0xb73409a4
retries = " No\000\002sý\000\001\000\000\000¸14·\b\a4·"
class2 = "\200ÈU\000¸14·è\0064·\222§I\000\200ÈU"
subclass2 = "(255?)\000\000\000\000\000\000@Åº\bô¯U"
class = 0xfdbcfd "CONTROL"
subclass = Variable "subclass" is not available.
By: Bobby Krupczak (rdk) 2005-08-22 20:37:43

Update.

I believe this problem is related to a problem I am having with using VoicePulse and iax2.

I debugged the crap out of iax_showframe() in channels/iax2-parse.c

I verified that all of the arguments to the first snprintf() seemed to
be valid and reasonable and I put if statements in to catch anything strange. Still, it crashed when debugging a call over iax2 to voicepulse.

Then, I doubled the space allocated to the tmp variable and the problem went away. Apparently, voicepulse is putting something in their iax2 packets that iax_showframe() is trying to print out and its overrunning the 256 bytes that tmp is originally allocated.

So, it may be worthwhile to put tmp[512] in the latest code.

I can gather more iax2 traces from voicepulse if this might help you guys better protect yourself from strange conditions. I dont know the iax2 protocol well enough to enough if what voicepulse is doing violates the protocol or not.

Thanks,

Bobby
By: Kevin P. Fleming (kpfleming) 2005-08-23 13:37:28

I have increaesed the buffer size in CVS HEAD and CVS v1-0. Thanks for the report and the debugging efforts!
By: Bobby Krupczak (rdk) 2005-08-23 16:46:23

I hate to be a pest about this ticket but . . .

During the course of troubleshooting with VoicePulse, I was able to get asterisk to crash again with iax2 debug enabled. However, instead of crashing every time, I can now only randomly get asterisk to crash. So, enlarging tmp[] to 512 bytes almost eliminates the problem.

Ultimately, though, there is either a protocol violation or some kind of crazy condition that can still occassionally crash asterisk if iax2 debug is enabled.

I dont know the protocol very well but I am happy to try to give you more trace data if you guys want to try to pursue this issue in more depth.

Thanks,

Bobby
By: Michael Jerris (mikej) 2005-08-23 17:10:42

ok, I know this is going to be a pain, but to effectively debug this further, we are going to need the following:
Run from current cvs head, built with valgrind, we will need a bt full, and a thread apply all bt from the core file.
A full iax debug and verbose from the call that is causing the crash.
An ethereal packet dump of the call that causes asterisk to crash (so we can see that last set of packets that you will not see in the debug due to the crash).

Thanks
By: Kevin P. Fleming (kpfleming) 2005-08-26 17:14:04

Let's approach this from a different direction; please use Ethereal or tcpdump to capture the packet stream between your Asterisk server and the provider. We can then use our own tools to look at the IAX2 packets and see what is going on with them.

If you need help with this process, find a bug marshal on #asterisk to help you.
By: Kevin P. Fleming (kpfleming) 2005-08-30 22:59:22

Any updates here? I want to get this fixed before this week's beta, but we can't make any progress without a packet capture.
By: Bobby Krupczak (rdk) 2005-08-31 15:11:23

Sorry for the delay. I'll try tonight to use tcpdump to capture a iax2
trace between my system and VP system.
By: Bobby Krupczak (rdk) 2005-08-31 20:48:13

I enabled iax2 debug, started tcpdump, and then dialed a number. Within
a fraction of a second, asterisk core dumps.

Bobby

asterisk*CLI> iax2 debug
IAX2 Debugging Enabled
-- Executing SetCallerID("SIP/rdkphone-bc1a", "6788820246") in new stack
-- Executing Dial("SIP/rdkphone-bc1a", "IAX2/Tnt81BDI3k:DTW88snr44@gwiaxt01.voicepulse.com/14042146014") in new stack
Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: NEW
Timestamp: 00012ms SCall: 00003 DCall: 00000 [66.234.228.160:4569]
VERSION : 2
CALLED NUMBER : 14042146014
CALLING NUMBER : 6788820246
LANGUAGE : en
USERNAME : Tnt81BDI3k
FORMAT : 4
CAPABILITY : 64518
ADSICPE : 2
DATE TIME : 186625362

-- Called Tnt81BDI3k:DTW88snr44@gwiaxt01.voicepulse.com/14042146014
Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: AUTHREQ
Timestamp: 00006ms SCall: 00315 DCall: 00003 [66.234.228.160:4569]
AUTHMETHODS : 2
CHALLENGE : 786831272
USERNAME : Tnt81BDI3k

Tx-Frame Retry[000] -- OSeqno: 001 ISeqno: 001 Type: IAX Subclass: AUTHREP
Timestamp: 00056ms SCall: 00003 DCall: 00315 [66.234.228.160:4569]
MD5 RESULT : 252dc2c3e133b91bb926f40daef2a104

Rx-Frame Retry[ No] -- OSeqno: 001 ISeqno: 002 Type: IAX Subclass: ACCEPT
Timestamp: 00044ms SCall: 00315 DCall: 00003 [66.234.228.160:4569]
FORMAT : 4

-- Call accepted by 66.234.228.160 (format ulaw)
-- Format for call is ulaw
Tx-Frame Retry[-01] -- OSeqno: 002 ISeqno: 002 Type: IAX Subclass: ACK
Timestamp: 00044ms SCall: 00003 DCall: 00315 [66.234.228.160:4569]
Rx-Frame Retry[ No] -- OSeqno: 002 ISeqno: 002 Type: CONTROL Subclass: (255?)
Timestamp: 00047ms SCall: 00315 DCall: 00003 [66.234.228.160:4569]
Tx-Frame Retry[-01] -- OSeqno: 002 ISeqno: 003 Type: IAX Subclass: ACK
Timestamp: 00047ms SCall: 00003 DCall: 00315 [66.234.228.160:4569]
-- IAX2/66.234.228.160:4569/3 stopped sounds
asterisk*CLI>
Disconnected from Asterisk server
[root@asterisk asterisk]# ps -aef | grep -i aster
root 1653 3867 0 21:42 pts/0 00:00:00 grep -i aster

(gdb) where
#0 0x0047372f in vfprintf () from /lib/libc.so.6
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
#2 0x00478be1 in snprintf () from /lib/libc.so.6
#3 0x0100a520 in iax_showframe (f=0x0, fhi=0xb737b9a4, rx=1, sin=0xb737e3a8,
datalen=0) at iax2-parser.c:444
#4 0x00fffeb4 in socket_read (id=0x8ff0008, fd=10, events=1, cbdata=0x0)
at chan_iax2.c:5157
ASTERISK-1 0x08052bd4 in ast_io_wait (ioc=0x9007238, howlong=0) at io.c:267
ASTERISK-2 0x00ff2953 in network_thread (ignore=0x0) at chan_iax2.c:6388
ASTERISK-3 0x005a7b80 in start_thread () from /lib/libpthread.so.0
ASTERISK-4 0x004ffdee in clone () from /lib/libc.so.6

(gdb) bt full
#0 0x0047372f in vfprintf () from /lib/libc.so.6
No symbol table info available.
#1 0x0049176f in vsnprintf () from /lib/libc.so.6
No symbol table info available.
#2 0x00478be1 in snprintf () from /lib/libc.so.6
No symbol table info available.
#3 0x0100a520 in iax_showframe (f=0x0, fhi=0xb737b9a4, rx=1, sin=0xb737e3a8,
datalen=0) at iax2-parser.c:444
frames = {0x1010975 "(0?)", 0x1010ad0 "DTMF ", 0x1010ad8 "VOICE ",
0x1010ae0 "VIDEO ", 0x100fd7d "CONTROL", 0x1010ae8 "NULL ",
0x1010af0 "IAX ", 0x1010af8 "TEXT ", 0x1010b00 "IMAGE ",
0x1010b08 "HTML ", 0x1010b10 "CNG "}
iaxs = {0x1010975 "(0?)", 0x10109b3 "NEW ", 0x10109bb "PING ",
0x10109c3 "PONG ", 0x10109cb "ACK ", 0x101097a "HANGUP ",
0x10109d3 "REJECT ", 0x10109db "ACCEPT ", 0x10109e3 "AUTHREQ",
0x10109eb "AUTHREP", 0x10109f3 "INVAL ", 0x10109fb "LAGRQ ",
0x1010a03 "LAGRP ", 0x1010a0b "REGREQ ", 0x1010a13 "REGAUTH",
0x1010a1b "REGACK ", 0x1010a23 "REGREJ ", 0x1010a2b "REGREL ",
0x1010a33 "VNAK ", 0x1010a3b "DPREQ ", 0x1010a43 "DPREP ",
0x1010a4b "DIAL ", 0x1010a53 "TXREQ ", 0x1010a5b "TXCNT ",
0x1010a63 "TXACC ", 0x1010a6b "TXREADY", 0x1010a73 "TXREL ",
0x1010a7b "TXREJ ", 0x1010a83 "QUELCH ", 0x1010a8b "UNQULCH",
0x1010a93 "POKE", 0x1010a98 "PAGE", 0x1010a9d "MWI",
0x1010aa1 "UNSUPPORTED", 0x1010aad "TRANSFER", 0x1010ab6 "PROVISION",
0x1010ac0 "FWDOWNLD", 0x1010ac9 "FWDATA"}
cmds = {0x1010975 "(0?)", 0x101097a "HANGUP ", 0x1010982 "RING ",
0x101098a "RINGING", 0x1010992 "ANSWER ", 0x101099a "BUSY ",
0x10109a2 "TKOFFHK ", 0x10109ab "OFFHOOK"}
fh = (struct ast_iax2_full_hdr *) 0xb737b9a4
retries = " No\000i\000\000\000\003\000\000\000ò\000\000\000\001\000\000"
class2 = "\200ÈU\000%\000\000\000*\000\000\000\025\000\000\000\037\000\000"
subclass2 = "(255?)\000·\215\\\026C¢\a\016\bô¯U"
class = 0x100fd7d "CONTROL"
subclass = Variable "subclass" is not available.
By: Kevin P. Fleming (kpfleming) 2005-09-02 16:09:56

The line numbers in your backtrace do not match any CVS Asterisk copies of iax2-parser.c that I can find. I will not be able to solve this problem without determining exactly what is happening on your system, and without a properly built Asterisk without optimization I can't do anything useful at all.

Your server appears to be dying while processing a control frame from VoicePulse indicating 'congestion', but there is no reason that should be happening. If you can provide remote access to this system at a time when it can be used for debugging, we can get it fixed. If not, we have no way to duplicate the problem and no way to get adequate debugging information from you, so we can't do much to help.
By: Bobby Krupczak (rdk) 2005-09-02 17:26:44

>The line numbers in your backtrace do not match any CVS Asterisk copies of >iax2-parser.c that I can find. I will not be able to solve this problem without

I'm using 1.0.9 that I modified to spit out debugging info and to help troubleshoot this problem.

Do you want me to email or attach my iax2_parser.c ? Or, do you want me to re-build with the vanilla 1.0.9 iax2_parser.c and re-tcpdump the core? (Sorry, I forgot that I had modified iax2_parser.c to try to fix the bug myself.)

>Your server appears to be dying while processing a control frame from >VoicePulse indicating 'congestion', but there is no reason that should be >happening.

VP has some sort of acknowledged problem (bug) and have claimed they are
working on it. Its been over a week. I dunno what the deal is on their end.

>If you can provide remote access to this system at a time when it can be used >or debugging, we can get it fixed. If not, we have no way to duplicate the >problem and no way to get adequate debugging information from you, so we can't >do much to help.

Email I received suggested that you guys had tools in house to take the trace and use it to core-dump your own systems. Is this true?

Alternatively, I could set up some sort of ssh into this system on a temporary basis. I'd like to do this only as a last resort. I think this bug is important to fix.

Let me know how to proceed.
By: Kevin P. Fleming (kpfleming) 2005-09-13 18:33:17

No, we cannot take the trace and core-dump our own systems... we can only inspect the packets and look for what (may) be wrong with them.

Please run the test with an _unmodified_ set of code from CVS v1-0 (not 1.0.9, there have been additions to the code base since 1.0.9 was released) so we have something to compare to on our end. If I can't see what line of code it is failing on, I can't help :-)
By: Mark Spencer (markster) 2005-10-04 13:49:38

I'm suspending this report since we haven't been able to get an updated backtrace.
By: Russell Bryant (russell) 2005-10-04 13:58:15

To check out the latest code from the version 1.0 branch of CVS, the command is "cvs co -r v1-0 asterisk".

If you need any help with this, feel free to contact me on IRC - 'drumkilla', or by email - russelb@clemson.edu.

Thanks!
By: Digium Subversion (svnbot) 2008-01-15 15:45:12.000-0600

Repository: asterisk
Revision: 6386

U trunk/channels/iax2-parser.c

------------------------------------------------------------------------
r6386 | kpfleming | 2008-01-15 15:45:12 -0600 (Tue, 15 Jan 2008) | 3 lines

ensure buffer is adequately sized for frames with lots of elements (issue ASTERISK-4847)
various minor formatting/cleanup changes

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=6386
By: Digium Subversion (svnbot) 2008-01-15 15:45:13.000-0600

Repository: asterisk
Revision: 6387

U branches/v1-0/channels/iax2-parser.c

------------------------------------------------------------------------
r6387 | kpfleming | 2008-01-15 15:45:13 -0600 (Tue, 15 Jan 2008) | 2 lines

ensure buffer is adequately sized for complex frames (issue ASTERISK-4847)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=6387