[Home]

Summary:ASTERISK-10918: IAX crashing Asterisk
Reporter:Thiago Garcia (thiagarcia)Labels:
Date Opened:2007-11-28 14:08:16.000-0600Date Closed:2008-02-18 16:06:39.000-0600
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Channels/chan_iax2
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) bt_full-97152.txt
( 1) thread_apply_all_bt-97152.txt
( 2) thread_apply_all_bt.txt
Description:After updated Asterisk from SVN-branch-1.4-r80601 to 1.4.14, my Asterisk in Server A is crashing, apparently this issue occurred when the call that outgoing across the IAX trunk in Server B is blind transfered in Server A between two IAX softphone.

I reverted  the Server A to SVN-branch-1.4-r80601.

Russell if you assign the bug for you, your login in the server is still active, else I can send debug file.

Asterisk-1.4.14
Zaptel-1.4.6
Libpri-1.4.2


****** ADDITIONAL INFORMATION ******

  -------------
Our scenario:
-------------
extensions prococol: IAX2
softphone: IDEFISK
softphone codec: ALAW/GSM/ULAW
* Server A are compiled with DONT_OPTIMIZE and DEBUG_THREADS options.
-------------

---------------------
Hardware Description:
---------------------

Server 'A':
-Dell PowerEdge 1800
-Dual Intel(R) Xeon(TM) CPU 3.00GHz Coreduo
-Digium TE110P card
-Digium TDM2400 (with 6 FXS modules)
-Digium TDM400P (with 4 FXO modules)

Server 'B':
-Dell PowerEdge 430SC
-Intel P4 2.4Ghz (hyperthreading enabled)
-Digium TE110P card
-Digium TDM400P (with 4 FXS modules)
---------------------
Comments:By: Tilghman Lesher (tilghman) 2007-12-03 16:39:32.000-0600

If you're reporting a crash, then we need you to follow the instructions in doc/backtrace.txt.
By: Thiago Garcia (thiagarcia) 2007-12-04 07:48:23.000-0600

Russell has access to the server and login opens automatically the command gdb in the core dump files.
If needed, I can send debug file.
By: Russell Bryant (russell) 2007-12-04 17:19:42.000-0600

I can't find the login information, but I'm sure it's still in my email somewhere.  What address would it have come from?
By: Thiago Garcia (thiagarcia) 2007-12-05 05:01:54.000-0600

The address is netadm@nexxera.com .
By: Russell Bryant (russell) 2007-12-05 11:08:37.000-0600

I logged in, but my user does not have access to the core files.
By: Thiago Garcia (thiagarcia) 2007-12-05 12:34:52.000-0600

Excuse Russell, I changeed the permissions now, please try now.
By: Russell Bryant (russell) 2007-12-05 13:26:12.000-0600

Neither backtrace really makes any sense.  They both look like:

#0  0xb6cd64d2 in socket_process (thread=0x81cc548) at chan_iax2.c:7405
7405                                    if (!iaxs[fr->callno]) {
(gdb) bt
#0  0xb6cd64d2 in socket_process (thread=0x81cc548) at chan_iax2.c:7405
#1  0xb6cdbd1c in socket_process (thread=0x81cc548) at chan_iax2.c:8143
#2  0x0810ff34 in dummy_start (data=0x81a04a0) at utils.c:843
#3  0xb7f811b3 in start_thread () from /lib/libpthread.so.0
#4  0xb7ecb03e in clone () from /lib/libc.so.6


However, socket_process() never calls itself, nor is it ever the first function in a thread as this backtrace shows.

Do you know if this backtrace was done on a version of Asterisk compiled without optimizations?  i.e. the DONT_OPTIMIZE setting?
By: Thiago Garcia (thiagarcia) 2007-12-05 13:40:14.000-0600

In the first time I compiled without DONT_OPTIMIZE and DEBUG_THREADS, when crashed the first time, I executed make clean and compiled with DONT_OPTIMIZE and DEBUG_THREADS, and the Asterisk crashes more two times.
By: Paul Crane (pcrane) 2007-12-10 20:58:10.000-0600

I've got the following error from a customer's machine, and was wondering if it's the same thing.

#0  0x02429066 in socket_process (thread=0xb7d29058) at chan_iax2.c:7525
#1  0x0242ea3e in iax2_process_thread (data=0xb7d29058) at chan_iax2.c:8309
#2  0x080f786b in dummy_start (data=0xb7d1abf8) at utils.c:852
#3  0x00c942fb in start_thread () from /lib/libpthread.so.0
#4  0x00bd793e in clone () from /lib/libc.so.6

This is from the following version of asterisk:

Asterisk SVN-branch-1.4-r91292, Copyright (C) 1999 - 2007 Digium, Inc. and others.

Running on a Fedora Core 7 linux distribution.


By: Matt Riddell (zx81) 2007-12-11 15:49:04.000-0600

I added the bt from our crash.  Let us know if you have any questions about it.
By: Matt Riddell (zx81) 2007-12-11 17:31:38.000-0600

have recompiled with dont optimize and will post bt once it crashes again
By: Thiago Garcia (thiagarcia) 2007-12-11 17:41:18.000-0600

Russell, I believe that "make clean" not cleaned the source, I am in vacation now.
I will return in january to work , I will remove the source before of compile once again for be sure that the source is clean.

Thank you.


By: Matt Riddell (zx81) 2007-12-11 20:15:37.000-0600

If its the same problem then we should have full bts soon.  :)
By: Matt Riddell (zx81) 2007-12-13 15:11:04.000-0600

Sorry, the customer was so angry we've ended up having to replace all phones with SIP phones and replace all outgoing IAX trunks with Zap analogue trunks.

We're still running with dont_optimize so if there is a crash we'll catch it, but its been two days now and no crash (expected as the crash seemed to be related to IAX2).

We do have other customers who are using IAX2 phones, but they are using Asterisk 1.4.11 from a tarball and so am hesitant to upgrade them.

I've just checked and they do have a couple of core dumps, so I'm enabling don't optimize and will see if its the same issue.

I've done a bt on the core from before I set the don't optimize on this customer's machine and it seems to be the same place:

#0  0x00e18c7b in socket_process (thread=0x88a04f8) at chan_iax2.c:7400
#1  0x00e1f005 in iax2_process_thread (data=0x88a04f8) at chan_iax2.c:8173
#2  0x080f407b in dummy_start (data=0x88a5ff8) at utils.c:775
#3  0x00cee2fb in start_thread () from /lib/libpthread.so.0
#4  0x00c3193e in clone () from /lib/libc.so.6
By: Tilghman Lesher (tilghman) 2008-01-07 15:15:04.000-0600

Okay, we're not getting anywhere with this.  Could one of you having this problem compile with the latest 1.4.17 and see if the problem persists?  Also, if you're able to reproduce with the latest code, please follow the instructions in doc/valgrind.txt.
By: Matt Riddell (zx81) 2008-01-07 23:50:45.000-0600

I've just checked the remaining customer with IAX2 phones (they're compiled with DONT_OPTIMIZE) and they've been up over a week, and there aren't any cores.  They may have been using the phones differently.  For myself I don't mind this issue being closed, I can just mail the dev list to get it reopened if I see it again.
By: Thiago Garcia (thiagarcia) 2008-01-09 11:41:59.000-0600

After upgrade to SVN-branch-1.4-r97152, the asterisk crash.

The bt full and thread apply all bt is Attached.

If you need access to machine, your login in the server is still active.

I do not know the same issue described is been previously.


By: Tilghman Lesher (tilghman) 2008-01-09 12:19:10.000-0600

thiagarcia:  please follow the instructions in doc/valgrind.txt.
By: Thiago Garcia (thiagarcia) 2008-01-09 15:09:03.000-0600

The valgrind crashes the machine "SO". I will create the tests PBX for this purpose.

The valgrind is same needed?
By: Thiago Garcia (thiagarcia) 2008-01-10 07:01:04.000-0600

Hello,

I did follow your instructions, but this version of Asterisk is very unstable here causing crashes and making the debug process a hell.
Valgrind is causing crashes too.

So, we'll downgrade to our last stable version SVN-branch-1.4-r80601.

Regards,

Thiago Garcia.


By: Thiago Garcia (thiagarcia) 2008-01-10 07:40:37.000-0600

I created a test enviroment machine. I copied the configuration archives of the production enviroment to the test machine. When i started the asterisk with valgrind it returned the following error:
(Observation: Im using AEL2)
But crashes occured only in production enviroment.

WARNING: Freeing unused memory at (nil), in ael_yyfree of ael_lex.c, line 2880
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089
WARNING: Freeing unused memory at (nil), in ast_yyfree of ast_expr2f.c, line 3089


By: Thiago Garcia (thiagarcia) 2008-01-10 09:01:49.000-0600

Russel can you intervene in this case?

Thank you.
By: jmls (jmls) 2008-02-06 04:22:09.000-0600

is this still an issue, or can we close
By: jmls (jmls) 2008-02-17 13:09:40.000-0600

last chance ...
By: Thiago Garcia (thiagarcia) 2008-02-18 15:25:33.000-0600

Jmls, I not upgraded anymore the asterisk.
The Valgrind crash the machine before the asterisk start or takes away huge load.
I sent the bt full and thread apply all.
I am the arrangement for help what will go necessary.

Thank you.
By: Tilghman Lesher (tilghman) 2008-02-18 16:06:39.000-0600

Reporter is unable to provide sufficient debugging information.  Cannot reproduce.  Closing.