[Home]

Summary:ASTERISK-10707: Manager API hangs on "Command: show channels"
Reporter:Alistair Cunningham (acunningham)Labels:
Date Opened:2007-11-07 07:02:27.000-0600Date Closed:2011-06-07 14:03:10
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/ManagerInterface
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) core_set_verbose_9.txt
( 1) coreShowLocks.txt
( 2) locks.txt
( 3) show_channels.txt
Description:If a client connects to the manager API, and does the following:

Action: Command
Command: show channels

That manager API session hangs indefinitely. Both "show channels" and "show channels concise" are affected, but other commands such as "core set verbose 9" are not. This affects both Asterisk 1.4.11 and 1.4.13, and possibly other versions as well.

****** ADDITIONAL INFORMATION ******

Will upload trace files shortly.
Comments:By: Alistair Cunningham (acunningham) 2007-11-07 07:07:17.000-0600

Files are uploaded; please let me know if you require any further information.
By: Alistair Cunningham (acunningham) 2007-11-07 07:31:47.000-0600

I've just tested a 1.4.5 system, and it's working correctly.
By: Alistair Cunningham (acunningham) 2007-11-07 08:15:10.000-0600

Further testing:

- "core show channels" and "core show channels concise" are also affected.

- Loading an Asterisk console with "asterisk -r", then doing "show channels" breaks all subsequent console commands on the 1.4.13 system I have been testing on:

test:~# asterisk -r
Asterisk 1.4.13, Copyright (C) 1999 - 2007 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 1.4.13 currently running on test (pid = 2785)
test*CLI> set verbose 9
Verbosity was 0 and is now 9
test*CLI> show channels
Channel              Location             State   Application(Data)            
SIP/111.111.111.111: (None)               Up      Bridged Call(Local/1234603@ens
Local/1234603@enswit 1234603@enswitch-pho Up      Dial(SIP/1234603@111.111.111.1
Local/1234603@enswit (None)               Up      Bridged Call(SIP/111.111.111.1
test*CLI> show channels
test*CLI> show channels
test*CLI> show channels
test*CLI> set verbose 9
test*CLI> show dialplan
test*CLI> exit

(hostname and IP address changed)

Note the lack of output from subsequent commands. Multiple "show dialplan" commands work correctly if no "show channels" has been run on this console. Exiting the console, then reloading it fixes the problem until the next "show channels".
By: Alistair Cunningham (acunningham) 2007-11-07 08:18:04.000-0600

Considering that the Asterisk console is also affected, Core-General may be a better category for this ticket.
By: Mark Michelson (mmichelson) 2007-11-07 10:07:05.000-0600

Are you able to reliably reproduce the problem? Are certain circumstances required to cause the behavior you're seeing? I just tried to reproduce and could not.
By: Alistair Cunningham (acunningham) 2007-11-07 10:13:23.000-0600

Yes, it always happens, and nothing special is required to trigger it.
By: Alistair Cunningham (acunningham) 2007-11-07 10:19:37.000-0600

After the last 2 notes, I did some further testing. I attempted to restart Asterisk on the machine I had been testing with - it had to be killed -9 to force it to quit. On restarting, the problem did not occur.

This system has a client connecting to the manager interface every 5 minutes to do a "show channels". Maybe the problem occurs only after a certain number of times. I will try again later.
By: Alistair Cunningham (acunningham) 2007-11-08 13:43:45.000-0600

The Asterisk 1.4.13 machine has now been running for 24 hours without problems. I will update the ticket if I have any more news.
By: Matt Riddell (zx81) 2007-11-15 20:54:09.000-0600

I've seen the same thing happen on SmoothTorque customer's machines.  We do a show channels concise every 5 seconds, but it seems to be pretty random as to when it stops responding.  We've ended up putting a timer, dropping the manager connection, reconnecting and it seems to be working ok.  Will post if I see it again.
By: Alistair Cunningham (acunningham) 2007-11-28 09:37:56.000-0600

We have now worked around this problem on the client end, so will probably not be in a position to provide any more information. This ticket can be closed or left open as Digium sees fit.
By: Denis Galvao (denisgalvao) 2007-11-28 13:30:39.000-0600

I have the same thing running 1.4.14. (.11 and .13 got the same results either)

After a core show channels, the system(CLI commands and the calls) didn't work anymore.

I had to do a kill -9 on the asterisk PID.

Something related to a deadlock in my opinion.
By: Jason Parker (jparker) 2007-12-12 18:39:04.000-0600

Has anybody managed to get the output of a `core show locks` while in this state?
By: Daniel Kühl Lima (dklima) 2007-12-12 19:27:28.000-0600

No, when CLI is on that stage, any command that I type have no result, and 'core show locks' trully lock CLI> prompt...
I'm just able to get core dump.

As http://bugs.digium.com/view.php?id=11080

Using Asterisk SVN-branch-1.4-r91890
By: Edoardo Serra - WeBRainstorm S.r.l. (webrainstorm) 2007-12-13 02:52:12.000-0600

IMHO this is related to http://bugs.digium.com/view.php?id=11413
My 0.02€
By: Tilghman Lesher (tilghman) 2008-01-04 10:09:50.000-0600

Since "core show locks" has now been fixed in the current version (1.4.17), I'd like to renew calls to get a "core show locks" when this problem occurs, if it still does.

If it no longer occurs, then we need to close this out.
By: Scott Cashon (scashon) 2008-01-15 11:08:42.000-0600

Have the same problem in 1.4.17.

We have a manager connection which runs frequent 'show channels concise' commands. After many of these (hundreds?) sometimes the commands timeout.

Once this happens output on the console is affected as well, but only once a 'show channels' command is issued. Other commands don't seem to be affected initially, but once a show channels is issued everything stops working. Call processing does not seem to be affected.

We can exit the console and go back in (asterisk -r) and the show channels will work exactly once. Subsequent show channels commands brings everything to a stop again:

phlast1*CLI> show channels concise
phlast1*CLI> set verbose 0
phlast1*CLI> set verbose 10
phlast1*CLI> show parkedcalls
phlast1*CLI> exit
[xxxx@phlast1 ~]# asterisk -r
Asterisk 1.4.17, Copyright (C) 1999 - 2007 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 1.4.17 currently running on phlast1 (pid = 6847)
phlast1*CLI> set verbose 0
Verbosity is now OFF
phlast1*CLI> show parkedcalls
Num                   Channel (Context         Extension    Pri ) Timeout
0 parked calls.
phlast1*CLI> core show channels concise
SIP/xxxxx-09bc62e0!xxxxx!2147!1!Ringing!AppDial!(Outgoing Line)!2147!xxxxxxxxxx_3101!3!0!(None)
SIP/xxxxx-b78289c0!xxxxx!2147!5!Ring!Dial!xxxxxxx| 20| tT!3101!0xxxxxxxxx!3!0!(None)
SIP/xxxxx-09977450!xxxxx!8188!1!Ringing!AppDial!(Outgoing Line)!8188!!3!0!(None)
SIP/xxxxx-b782a010!xxxxx!8188!22!Ring!Dial!SIP/xxxxxxxx| 24|t!xxxxxx!!3!0!(None)

phlast1*CLI> core show channels concise
phlast1*CLI> set verbose 10
phlast1*CLI> show parked calls
phlast1*CLI>

[ Showing console commands not working, exiting and then commands working (including show channels, once) until a show channels is issued. ]

I am able to do a 'core show locks', I'll upload that next...
tsc
By: Mark Michelson (mmichelson) 2008-01-15 13:39:53.000-0600

The core show locks output you have provided shows the same deadlock experienced in issue ASTERISK-11200. I uploaded a patch there (called 11730.patch). Please try using that and see if that fixes the issue.
By: Scott Cashon (scashon) 2008-01-15 13:53:59.000-0600

Thanks, we'll apply that patch and see how it goes...

tsc
By: Mark Michelson (mmichelson) 2008-01-16 11:22:44.000-0600

Since this is the same deadlock as reported in issue 11730 and the reporter there has tested the patch and it works for him, I have commmitted the patch to 1.4 in revision 98964 and to trunk in revision 98965. I am closing this issue since it is fixed now. Thanks for the help in getting this resolved.
By: Alistair Cunningham (acunningham) 2008-02-26 14:44:48.000-0600

This still seems to be happening on 1.4.18:

[root@test ~]# asterisk -r
Asterisk 1.4.18, Copyright (C) 1999 - 2008 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 1.4.18 currently running on test (pid = 2338)
test*CLI> show channels
test*CLI> exit

Further traces available on request.

By: dovid (dovid) 2008-02-27 12:49:50.000-0600

Spoke to russelb on IRC. He would like an output of 'core show locks'. (Make sure build has  'DEBUG_THREADS' turned on).
By: Mark Michelson (mmichelson) 2008-03-18 16:03:02

Ping.

Can we see the "core show locks" output as Dovid requested? Thanks!
By: Alistair Cunningham (acunningham) 2008-03-18 16:11:36

I'm afraid Dovid's machine has been downgraded to 1.2 so we cannot run this. If we manage to reproduce the problem on another machine, I can upload the output from this command. In the meantime, can we please leave the ticket open?
By: Mark Michelson (mmichelson) 2008-03-18 16:35:53

Considering there are a couple of other deadlock issues open with regards to core show channels, and we have no idea whether this one corresponds to one of the other reported ones, I'm going to suspend this. Don't hesitate to re-open if you're able to reproduce it again and provide "core show locks" output.
By: Alistair Cunningham (acunningham) 2008-03-27 11:58:03

This has happened again on a different system running 1.4.18.1, but core show locks does not seem to exist. Maybe we need different compile options?

test*CLI> core show locks
No such command 'core show' (type 'help' for help)

We're also seeing this on the console:

[Mar 27 16:58:41] ERROR[26860]: chan_sip.c:15385 sipsock_read: We could NOT get the channel lock for SIP/1.2.3.4-c0002120!
[Mar 27 16:58:41] ERROR[26860]: chan_sip.c:15386 sipsock_read: SIP transaction failed: 3c26ad5a3d09-8ahqbt5fjjs2@elmegIP290
[Mar 27 16:58:41] ERROR[26860]: chan_sip.c:15385 sipsock_read: We could NOT get the channel lock for SIP/1.2.3.4-c40601b0!
[Mar 27 16:58:41] ERROR[26860]: chan_sip.c:15386 sipsock_read: SIP transaction failed: 3339600196@1.2.3.4

IP addresses have been replaced with 1.2.3.4. Asterisk has now been restarted and the problem has gone away for now.

By: Daniel Morgan (danm) 2008-03-27 12:58:43

This same problem has occurred several time over the last two weeks.
By: Alistair Cunningham (acunningham) 2008-03-27 13:12:00

Ok, we have this machine compiled with DEBUG_THREADS so core show locks works. I will update the ticket once the problem occurs again.
By: Joshua C. Colp (jcolp) 2008-04-14 13:32:58

Do you have the core show locks yet?
By: Alistair Cunningham (acunningham) 2008-04-14 13:46:47

No, the system owner decided to downgrade to 1.2.27 rather than wait for the problem to occur again. I will let you know if we manage to get it.
By: Joshua C. Colp (jcolp) 2008-04-14 13:55:19

Alrighty, I'll suspend this for now.
By: Alistair Cunningham (acunningham) 2008-05-05 17:23:18

We now have a "core show locks". This is from a different machine affected by the same problem. Please see the attached locks.txt file.


By: Mark Michelson (mmichelson) 2008-05-07 13:30:33

What version of Asterisk is that core show locks from? It looks like it's not very recent.

I hate to ask this since I know this is a rarely occurring issue, but if this installation isn't up-to-date, could you please update it to either 1.4.20 or the latest svn revision of 1.4 and be sure this is still a problem? If I'm mistaken and this was taken from a recent 1.4 checkout, let me know and I'll take a closer look.
By: Alistair Cunningham (acunningham) 2008-05-07 13:32:50

It was taken from a 1.4.18 system. I'm afraid it won't be possible to try anything later as the customer has downgraded to 1.2, and is most unlikely to agree to upgrade again until this problem is fixed.
By: Mark Michelson (mmichelson) 2008-05-15 11:27:35

Well, it sounds like we have a catch 22 here. Your client won't upgrade to a later version until this problem has been fixed, but we can't tell if it's actually been fixed because we have no data from an attempt at using the upgraded installation. I hate having to do this, but I can't do anything other than suspend this issue until I can get confirmation that this issue still occurs on the latest 1.4.