[Home]

Summary:ASTERISK-11010: Asterisk 1.4.15 uses 200% of CPU randomly and crashes the machine.
Reporter:Ygor Lemos (ygor)Labels:
Date Opened:2007-12-11 06:06:26.000-0600Date Closed:2011-06-07 14:00:32
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:I have several Asterisk servers running all over the country and when I upgraded those servers, they all started to randomly crash.

The unique thing I have detected is that on Core 2 Duo machines the CPU Usage goes around 200% CPU on top and on Xeon Quad-Core machines I get around 360% usage of cpu, causing random crashes and completely inutilising the machine.

Asterisk runs normally and this is a totaly random effect.

****** ADDITIONAL INFORMATION ******

Asterisk 1.4.15 built by ygor @ ipbx on a i686 running Linux on 2007-12-10 04:19:19 UTC
The 'show version' command is deprecated and will be removed in a future release. Please use 'core show version' instead.

----

Linux ezipbx 2.6.22-3-686 #1 SMP Mon Nov 12 08:32:57 UTC 2007 i686 GNU/Linux


Using: Asterisk 1.4.15, Zaptel 1.4.7, Asterisk-addons 1.4.5 and libpri 1.4.2.

Libpri installed from Debian Lenny APT repositories as the current libc6 update breaks libpri compilation (which must be reported on another bug...)

Comments:By: Ygor Lemos (ygor) 2007-12-11 09:32:00.000-0600

My guess is that it has something to do with MySQL integration or with SIP packetization....

Ps.: MySQL password is correct, and everything works on database... It just gets mad before the crash....

Im sending the asterisk log, just before one of those weird Asterisk deadlocks:



[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: mysql_real_connect(mysql,localhost,root,dbpass,asterisk,...) failed
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Identifier 0, identifier_type 1 not found in identifier list
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: aMYSQL_query: Invalid connection identifier 0 passed in aMYSQL_query
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Identifier 0, identifier_type 1 not found in identifier list
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: aMYSQL_query: Invalid connection identifier 0 passed in aMYSQL_query
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Identifier 0, identifier_type 2 not found in identifier list
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: aMYSQL_fetch: Invalid result identifier 0 passed
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Identifier 0, identifier_type 2 not found in identifier list
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Invalid result identifier 0 passed in aMYSQL_clear
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Identifier 0, identifier_type 1 not found in identifier list
[Dec 11 13:23:51] WARNING[18976] app_addon_sql_mysql.c: Invalid connection identifier 0 passed in aMYSQL_disconnect
[Dec 11 13:23:51] WARNING[18976] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:23:56] WARNING[18982] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:23:57] WARNING[18982] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:03] WARNING[18983] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:03] WARNING[18983] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:04] WARNING[18984] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:04] WARNING[18984] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:04] WARNING[18984] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission b1a7101f-15e51bf8-f3785e85@10.0.0.185 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission c411eec7-449e391c-76ad550d@10.0.0.170 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Hanging up call c411eec7-449e391c-76ad550d@10.0.0.170 - no reply to our critical packet.
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission b1a7101f-15e51bf8-f3785e85@10.0.0.185 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission c411eec7-449e391c-76ad550d@10.0.0.170 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Hanging up call c411eec7-449e391c-76ad550d@10.0.0.170 - no reply to our critical packet.
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission b1a7101f-15e51bf8-f3785e85@10.0.0.185 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission c411eec7-449e391c-76ad550d@10.0.0.170 for seqno 2 (Critical Response)
[Dec 11 13:24:24] WARNING[18097] chan_sip.c: Hanging up call c411eec7-449e391c-76ad550d@10.0.0.170 - no reply to our critical packet.
[Dec 11 13:24:28] WARNING[18990] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:29] WARNING[18990] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:29] WARNING[18990] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:32] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission d382480d-e61e7304-4155ec6f@10.0.0.162 for seqno 2 (Critical Response)
[Dec 11 13:24:32] WARNING[18097] chan_sip.c: Hanging up call d382480d-e61e7304-4155ec6f@10.0.0.162 - no reply to our critical packet.
[Dec 11 13:24:32] WARNING[18097] chan_sip.c: Maximum retries exceeded on transmission d382480d-e61e7304-4155ec6f@10.0.0.162 for seqno 2 (Critical Response)
[Dec 11 13:24:32] WARNING[18097] chan_sip.c: Hanging up call d382480d-e61e7304-4155ec6f@10.0.0.162 - no reply to our critical packet.
[Dec 11 13:24:37] NOTICE[18097] chan_sip.c: Peer '6107' is now Reachable. (9ms / 2000ms)
[Dec 11 13:24:43] WARNING[18993] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:44] WARNING[18993] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:24:44] NOTICE[18097] chan_sip.c: Peer '6144' is now Reachable. (9ms / 2000ms)
[Dec 11 13:24:45] NOTICE[18097] chan_sip.c: Peer '6150' is now Reachable. (16ms / 2000ms)
[Dec 11 13:24:45] NOTICE[18097] chan_sip.c: Peer '6119' is now Reachable. (10ms / 2000ms)
[Dec 11 13:24:45] NOTICE[18097] chan_sip.c: Peer '6177' is now Reachable. (15ms / 2000ms)
[Dec 11 13:24:48] NOTICE[18097] chan_sip.c: Peer '6106' is now Reachable. (14ms / 2000ms)
[Dec 11 13:24:50] NOTICE[18097] chan_sip.c: Peer '6116' is now Reachable. (14ms / 2000ms)
[Dec 11 13:24:56] NOTICE[18097] chan_sip.c: Peer '6102' is now Reachable. (8ms / 2000ms)
[Dec 11 13:24:59] NOTICE[18097] chan_sip.c: Peer '6140' is now Reachable. (12ms / 2000ms)
[Dec 11 13:25:01] NOTICE[18097] chan_sip.c: Peer '6123' is now Reachable. (13ms / 2000ms)
[Dec 11 13:25:12] WARNING[18995] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:25:12] WARNING[18995] app_realtime.c: The RealTime application has been deprecated in favor of the REALTIME dialplan function.
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6141' is now UNREACHABLE!  Last qualify: 13
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6114' is now UNREACHABLE!  Last qualify: 12
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6104' is now UNREACHABLE!  Last qualify: 14
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6122' is now UNREACHABLE!  Last qualify: 14
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6127' is now UNREACHABLE!  Last qualify: 12
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6124' is now UNREACHABLE!  Last qualify: 13
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6108' is now UNREACHABLE!  Last qualify: 14
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6126' is now UNREACHABLE!  Last qualify: 13
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6129' is now UNREACHABLE!  Last qualify: 12
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6134' is now UNREACHABLE!  Last qualify: 13
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6113' is now UNREACHABLE!  Last qualify: 13
[Dec 11 13:25:37] NOTICE[18097] chan_sip.c: Peer '6142' is now UNREACHABLE!  Last qualify: 14
By: Tilghman Lesher (tilghman) 2007-12-11 09:51:31.000-0600

If you're going to report a CRASH, you need to follow the instructions in doc/backtrace.txt
By: Ygor Lemos (ygor) 2007-12-11 11:19:33.000-0600

The problem is that Asterisk completely inutilize the server before generate any core dumps... this is why i didn't attach any bt or full bt here.

If I can collect ANY information to help the development team to solve this issues with the latest version of Asterisk, please tell me how...

I have collected information with several other people having the same troubles in a variety of hardware and software configurations.

Could this be related to the new libc6 ?

I did use Debian Lenny (testing), but I heard from people using etch (stable), having the same problems.... random lockups, full CPU ( all cores ) utilization, machine hangs, etc...

Nobody gets any core dumps.
By: Ygor Lemos (ygor) 2007-12-11 11:36:06.000-0600

Does a valgrind with a malloc debug report helps ?!
By: Tilghman Lesher (tilghman) 2007-12-12 11:25:07.000-0600

Yes, a valgrind output would help.
By: Ygor Lemos (ygor) 2007-12-13 04:58:25.000-0600

Ok, I'll send you a Valgrind ASAP...

For now, to reproduce the problem, just follow:

* Install Debian Lenny ( some partners reported that the error occurs on etch too... )
* apt-get update
* apt-get dist-upgrade (from Debian official repositories)
* apt-get the necessary packages for Asterisk compilation...
* Install  the libpri from Debian repository (libpri-dev, libpri1.0) ( as the current libpri doesn't compile anymore and no official correction has been released....)
* Compile zaptel latest
* Compile Asterisk 1.4.15 and Asterisk-addons latest
* Run Asterisk with a Zaptel hardware (tested with wcte12xp)
* Keep some calls running on the server with MixMonitor for some hours (don't know if without MixMonitor the error reproduces as I've tested it with MixMonitor only...)
* Asterisk will hang soon...
By: Tilghman Lesher (tilghman) 2007-12-27 17:45:15.000-0600

I need a valgrind output with the latest version or this report will be closed.
By: Tilghman Lesher (tilghman) 2008-01-07 15:44:51.000-0600

No response from reporter.