[Home]

Summary:ASTERISK-10924: Strange freezing of the Manager and Asterisk Console
Reporter:Antonis Psaras (apsaras)Labels:
Date Opened:2007-11-29 04:03:07.000-0600Date Closed:2007-12-24 10:22:56.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20071203__bug11413.diff.txt
( 1) showlocks1137.txt
( 2) showlocks-20071207.txt
( 3) showlocks200712240017.txt
Description:We upgrade from 1.4.13 to 1.4.14 before a few days and we face the following problem.

After a few hours running smoothly the manager API does not respond and the asterisk console is freezing. Tab completion is not working and some commands does not return results (like show channels).

On the system there are 3 PRI lines on a TE410P and the strange thing is that the third line is working fine for incoming and outgoing but the first line is always returns busy in both incoming and outgoing.

The problem is repeated each day more than 4 times and is resolved with asterisk restart.


****** ADDITIONAL INFORMATION ******

The system is based on
Dell 1950 Xeon Dual Core
RedHat ES 64bit
Asterisk 1.4.14
Zaptel 1.4.6

We have 3 PRI with the following config
1st Slave/CPE (clock 1)
2nd Master/NET
3rd Slave/CPE (clock 2)

On the system we have 230 Snom SIP Phone Registered and the concurrent calls are limited to 90.
Comments:By: Russell Bryant (russell) 2007-11-29 07:38:09.000-0600

Please compile with DEBUG_THREADS enabled (run "make menuselect", go to the "Compiler Flags" section, enable it by hitting space, hit 'x' to save and quit).

Then, when this happens, please try to grab the output of the "core show locks" CLI command.

# asterisk -rx "core show locks" >showlocks.txt
By: Antonis Psaras (apsaras) 2007-11-30 04:11:12.000-0600

Please check the attached file for the locks.
By: Russell Bryant (russell) 2007-11-30 12:25:34.000-0600

Perfect.  That's exactly what we needed.  Thanks for the quick response.
By: Tilghman Lesher (tilghman) 2007-12-03 10:30:22.000-0600

This should fix it.  Please test and report back.
By: Antonis Psaras (apsaras) 2007-12-04 08:40:10.000-0600

We had the same problem again but now we couldn't even execute the show locks command. Executing asterisk -rv we didn't get the CLI>. Is there any other way to debug this?
By: Tilghman Lesher (tilghman) 2007-12-04 09:17:19.000-0600

apsaras:  are you getting this freezing, even with this patch applied?
By: Antonis Psaras (apsaras) 2007-12-04 10:07:25.000-0600

yes
By: Antonis Psaras (apsaras) 2007-12-07 09:03:38.000-0600

We managed to get one more showlock file.
By: Antonis Psaras (apsaras) 2007-12-07 10:44:39.000-0600

We 've tried 1.4.15 but we get deadlock during sip registration at the initialization of asterisk.
By: Tilghman Lesher (tilghman) 2007-12-07 11:01:31.000-0600

apsaras:  please upgrade to SVN revision 91074 or later and test again.
By: Antonis Psaras (apsaras) 2007-12-09 15:44:42.000-0600

We tried SVN branch 1.4 rev 91983 and after the installation we get the following error messages on the console

[Dec  9 23:40:46] ERROR[23105]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:342 __ast_pthread_mutex_lock: pbx.c line 2014 (ast_hint_state_changed): '&conlock' was locked here.
[Dec  9 23:40:46] ERROR[23105]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:338 __ast_pthread_mutex_lock: pbx.c line 1803 (pbx_extension_helper): Deadlock? waited 120 sec for mutex '&conlock'?
[Dec  9 23:40:46] ERROR[23105]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:342 __ast_pthread_mutex_lock: pbx.c line 2014 (ast_hint_state_changed): '&conlock' was locked here.
[Dec  9 23:40:46] ERROR[23105]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:338 __ast_pthread_mutex_lock: pbx.c line 1803 (pbx_extension_helper): Deadlock? waited 120 sec for mutex '&conlock'?
[Dec  9 23:40:47] ERROR[23105]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:342 __ast_pthread_mutex_lock: pbx.c line 2014 (ast_hint_state_changed): '&conlock' was locked here.
[Dec  9 23:40:48] ERROR[23103]: /usr/src/microbase/asterisk-1.4/svn/branches/asterisk-1.4/include/ast:338 __ast_pthread_mutex_lock: pbx.c line 6115 (ast_lock_contexts): Deadlock? waited 135 sec for mutex '&conlock'?

so we rolle back to 1.4.14

Any suggestion?
By: Edoardo Serra - WeBRainstorm S.r.l. (webrainstorm) 2007-12-12 03:01:52.000-0600

We're having the same issue with 1.4.15 when Asterisk is trying to traverse the channels list (it happens only after a while Asterisk is running)

Issuing a "core show channels" once per CLI session works properly but it doesn't display the summary information at the end (N active channels / N active calls)
At that point the console is freezed

Issuing a "core show channel <TAB>" can also freeze the console.

We have a Sangoma A104D with 3x E1

I moved production on another server so I can get more debugging information about that.
By: Eduard M. (callmewind) 2007-12-17 05:21:17.000-0600

I can confirm this bug too. We have a java application attached to Manager API. It issues a show channels every 2 seconds and Asterisk stop responding randomly (but only when there are active channels).
Sorry but now I can't add debug info.
By: ibercom (ibercom) 2007-12-17 06:53:53.000-0600

We're having the same issue with 1.4.15. It happens only after a while Asterisk is running with calls (Zap Channels) aprox. 1 or 2 hours. If Asterisk don't proccess calls then it's working several days.

With 1.4.13 no problems.
By: Marty Riedling (mariedling) 2007-12-21 13:32:27.000-0600

I too am having this problem with 1.4.14 and 1.4.15. I upgraded to 1.4.16 last night and seem to be running ok so far. I will post back next week If this appears to have fixed the issue. I was seeing the issue every afternoon, so we will see.
By: Antonis Psaras (apsaras) 2007-12-23 16:31:35.000-0600

Tested with 1.4.16.2 and I have the same problem. I am attaching lock file showlocks200712240017.txt
By: Matt Riddell (zx81) 2007-12-24 03:55:08.000-0600

This is a pretty major problem, and it has affected one of our customers again recently.  Anything anyone can provide that would assist in debugging this?
By: Marty Riedling (mariedling) 2007-12-24 07:27:32.000-0600

Ours locked up again on Saturday. So 1.4.16 does not fix this issue. However if I downgrade to 1.4.13 I still have the issue.
By: Tilghman Lesher (tilghman) 2007-12-24 09:31:05.000-0600

To be clear, are you testing WITH the patch above?  I know it's not fixed in the release.
By: Tilghman Lesher (tilghman) 2007-12-24 10:06:02.000-0600

apsaras:  the deadlock you're experiencing in that output should be resolved in revision 94466.
By: Digium Subversion (svnbot) 2007-12-24 10:13:40.000-0600

Repository: asterisk
Revision: 94765

U   branches/1.4/channels/chan_sip.c

------------------------------------------------------------------------
r94765 | tilghman | 2007-12-24 10:13:39 -0600 (Mon, 24 Dec 2007) | 5 lines

More deadlock avoidance code (this time between sip_monitor and sip_hangup)
Reported by: apsaras
Patch by: tilghman
(Closes issue ASTERISK-10924)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=94765
By: Digium Subversion (svnbot) 2007-12-24 10:22:56.000-0600

Repository: asterisk
Revision: 94766

_U  trunk/

------------------------------------------------------------------------
r94766 | tilghman | 2007-12-24 10:22:55 -0600 (Mon, 24 Dec 2007) | 12 lines

Blocked revisions 94765 via svnmerge

........
r94765 | tilghman | 2007-12-24 10:17:01 -0600 (Mon, 24 Dec 2007) | 5 lines

More deadlock avoidance code (this time between sip_monitor and sip_hangup)
Reported by: apsaras
Patch by: tilghman
(Closes issue ASTERISK-10924)

........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=94766