[Home]

Summary:ASTERISK-10007: "Mute" IAX2 Extensions
Reporter:Thiago Garcia (thiagarcia)Labels:
Date Opened:2007-08-01 15:44:41Date Closed:2007-08-23 12:12:40
Priority:BlockerRegression?No
Status:Closed/CompleteComponents:Channels/chan_iax2
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) core-show-locks.txt
( 1) core-show-locks-2.txt
Description:In last two weeks, we've got two problems on IAX2 extensions. Our iax extensions stop responding and it's not possible make and receive calls.
The iaxthreadcount and iaxmaxthreadcount parameters are set to '256'.

In messeges log:

[Aug  1 17:31:26] NOTICE[13268] chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!
[Aug  1 17:31:26] NOTICE[13267] chan_iax2.c: Out of idle IAX2 threads for scheduling!
[Aug  1 17:31:27] NOTICE[13268] chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!
[Aug  1 17:31:27] NOTICE[13267] chan_iax2.c: Out of idle IAX2 threads for scheduling!
[Aug  1 17:31:28] NOTICE[13268] chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!
[Aug  1 17:31:28] NOTICE[13267] chan_iax2.c: Out of idle IAX2 threads for scheduling!
[Aug  1 17:31:29] NOTICE[13268] chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!
[Aug  1 17:31:29] NOTICE[13267] chan_iax2.c: Out of idle IAX2 threads for scheduling!
[Aug  1 17:31:30] NOTICE[13268] chan_iax2.c: Out of idle IAX2 threads for I/O, pausing!
[Aug  1 17:31:30] NOTICE[13267] chan_iax2.c: Out of idle IAX2 threads for scheduling!

The issue is similar of bug ASTERISK-9559614, but the core is different.

Russell if you assign the bug for you, your login in the server is still active, else I can send debug file.


****** ADDITIONAL INFORMATION ******

-------------
Our scenario:
-------------
extensions prococol: IAX2
softphone: IDEFISK
softphone codec: ALAW/ULAW/GSM
* Both are compiled with DONT_OPTIMIZE and DEBUG_THREADS options.
-------------
Comments:By: Russell Bryant (russell) 2007-08-01 15:52:11

I will use the login I have to take a look at the backtrace information as soon as I can.  It may be a couple days before I get to it ...
By: Thiago Garcia (thiagarcia) 2007-08-01 15:55:12

Ok, thank you.
If you need any information, ask me.
By: Russell Bryant (russell) 2007-08-03 15:14:42

It looks like my login isn't set up correctly for this one.  I am disconnected immediately after entering my password.
By: Thiago Garcia (thiagarcia) 2007-08-03 15:19:39

I'm sorry, the shell was wrong. I corrected it now.
By: Russell Bryant (russell) 2007-08-03 17:23:14

If you can get this to happen with the newest code in the 1.4 branch, please post the output of the "core show locks" CLI command.
By: Thiago Garcia (thiagarcia) 2007-08-04 22:11:57

I did the upgrade.
If I have any issue I will post the output of the "core show locks" CLI command.
By: Thiago Garcia (thiagarcia) 2007-08-10 09:18:06

I upload the file containing the command core show locks, now, my asterisk is SVN-branch-1.4-r78242.
By: Russell Bryant (russell) 2007-08-10 10:39:25

Yikes ... your output made me realize that I messed something up with "core show locks".  If you update, the output should be a lot more useful.
By: Russell Bryant (russell) 2007-08-10 13:51:42

Also, be sure to increase the size of your terminal buffer, or just tee the output of the asterisk console to a file.  The output of "core show locks" on your system is going to be extremely long, as it has over 500 threads in the core dumps I have looked through.
By: Thiago Garcia (thiagarcia) 2007-08-10 14:06:57

I changed my terminal buffer to infinite.
Today I will upgrade the asterisk because the change in lock.h. To reproduce the error earlier, I will change the iaxthreadcount and iaxmaxthreadcount to defaults.
Thank you.
By: Russell Bryant (russell) 2007-08-13 15:31:24

I just fixed a lot of deadlock situations in chan_iax2 in the 1.4 branch.  This should fix your problem.  However, if it doesn't, please reopen this issue.  Thanks!
By: Thiago Garcia (thiagarcia) 2007-08-14 08:58:36

The issue continues, my asterisk is SVN-branch-1.4-r79301.
The output of command core show locks attached in core-show-locks-2.txt.
I not generated the core, the asterisk died before.


By: Digium Subversion (svnbot) 2007-08-16 16:11:34

Repository: asterisk
Revision: 79756

------------------------------------------------------------------------
r79756 | russell | 2007-08-16 16:11:34 -0500 (Thu, 16 Aug 2007) | 11 lines

Fix more deadlocks in chan_iax2 that were introduced by making frame handling
and scheduling multi-threaded.  Unfortunately, we have to do some expensive
deadlock avoidance when queueing frames on to the ast_channel owner of the IAX2
pvt struct.  This was already handled for regular frames, but ast_queue_hangup
and ast_queue_control were still used directly.  Making these changes introduced
even more places where the IAX2 pvt struct can disappear in the context of a
function holding its lock due to calling a function that has to unlock/lock it
to avoid deadlocks.  I went through and fixed all of these places to account for
this possibility.
(issue ASTERISK-10007, patch by me)

------------------------------------------------------------------------
By: Digium Subversion (svnbot) 2007-08-16 16:15:48

Repository: asterisk
Revision: 79764

------------------------------------------------------------------------
r79764 | russell | 2007-08-16 16:15:48 -0500 (Thu, 16 Aug 2007) | 19 lines

Merged revisions 79756 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r79756 | russell | 2007-08-16 16:29:24 -0500 (Thu, 16 Aug 2007) | 11 lines

Fix more deadlocks in chan_iax2 that were introduced by making frame handling
and scheduling multi-threaded.  Unfortunately, we have to do some expensive
deadlock avoidance when queueing frames on to the ast_channel owner of the IAX2
pvt struct.  This was already handled for regular frames, but ast_queue_hangup
and ast_queue_control were still used directly.  Making these changes introduced
even more places where the IAX2 pvt struct can disappear in the context of a
function holding its lock due to calling a function that has to unlock/lock it
to avoid deadlocks.  I went through and fixed all of these places to account for
this possibility.
(issue ASTERISK-10007, patch by me)

........

------------------------------------------------------------------------
By: Russell Bryant (russell) 2007-08-16 16:16:36

Give it another try after that last commit.
By: Digium Subversion (svnbot) 2007-08-16 20:05:54

Repository: asterisk
Revision: 79825

------------------------------------------------------------------------
r79825 | file | 2007-08-16 20:05:53 -0500 (Thu, 16 Aug 2007) | 106 lines

Merged revisions 79747,79749,79755,79764,79788,79794,79813,79824 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r79747 | tilghman | 2007-08-16 18:09:46 -0300 (Thu, 16 Aug 2007) | 2 lines

Don't reload a configuration file if nothing has changed.

................
r79749 | mmichelson | 2007-08-16 18:21:35 -0300 (Thu, 16 Aug 2007) | 16 lines

Merged revisions 79748 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r79748 | mmichelson | 2007-08-16 16:16:40 -0500 (Thu, 16 Aug 2007) | 8 lines

Fixes a problem where agents would get stuck busy due to their wrapuptime being longer than the queue's wrapuptime and
ringinuse=no for the queue.

(closes issue ASTERISK-9889, reported by Doug, repaired by me)

Special thanks to fkasumovic for pointing out the source of the problem and to bweschke for helping to come up with a solution!


........

................
r79755 | file | 2007-08-16 18:28:50 -0300 (Thu, 16 Aug 2007) | 2 lines

Fix properties on trunk again.

................
r79764 | russell | 2007-08-16 18:33:38 -0300 (Thu, 16 Aug 2007) | 19 lines

Merged revisions 79756 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r79756 | russell | 2007-08-16 16:29:24 -0500 (Thu, 16 Aug 2007) | 11 lines

Fix more deadlocks in chan_iax2 that were introduced by making frame handling
and scheduling multi-threaded.  Unfortunately, we have to do some expensive
deadlock avoidance when queueing frames on to the ast_channel owner of the IAX2
pvt struct.  This was already handled for regular frames, but ast_queue_hangup
and ast_queue_control were still used directly.  Making these changes introduced
even more places where the IAX2 pvt struct can disappear in the context of a
function holding its lock due to calling a function that has to unlock/lock it
to avoid deadlocks.  I went through and fixed all of these places to account for
this possibility.
(issue ASTERISK-10007, patch by me)

........

................
r79788 | russell | 2007-08-16 19:30:39 -0300 (Thu, 16 Aug 2007) | 22 lines

Merged revisions 79778 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r79778 | russell | 2007-08-16 17:24:25 -0500 (Thu, 16 Aug 2007) | 14 lines

This patch fixes a bug where reloading the module with "module reload" did not
delete classes from memory that were no longer in the config.  This patch fixes
that problem as well as another one.  Previously, if you reloaded MOH using the
"moh reload" CLI command, which behaved differently than "module reload ...",
MOH had to be stopped on every channel and started again immediately.  However,
there was no way to tell what class was being used, so they would all fall back
to the default class.

(closes issue ASTERISK-9825)
Reported by: blitzrage
Patches:
     asterisk-10139-advanced.diff.txt uploaded by jamesgolovich (license 176)
Tested by: jamesgolovich

........

................
r79794 | russell | 2007-08-16 19:33:02 -0300 (Thu, 16 Aug 2007) | 12 lines

Merged revisions 79792 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r79792 | russell | 2007-08-16 17:32:33 -0500 (Thu, 16 Aug 2007) | 4 lines

Fix a little race condition that could cause a crash if two channels had MOH
stopped at the same time that were using a class that had been marked for
deletion when its use count hits zero.

........

................
r79813 | tilghman | 2007-08-16 20:31:14 -0300 (Thu, 16 Aug 2007) | 2 lines

Revise dialplan locks to permit multiple locks per channel, but with deadlock avoidance

................
r79824 | file | 2007-08-16 22:19:04 -0300 (Thu, 16 Aug 2007) | 2 lines

Fix building of chan_zap under development mode without libpri and libss7 installed.

................

------------------------------------------------------------------------
By: Russell Bryant (russell) 2007-08-23 12:12:40

I am assuming this is now fixed.  Feel free to reopen or file a new issue with "core show locks" output if the problem persists.