[Home]

Summary:ASTERISK-01012: pbx_spool
Reporter:mlh (mlh)Labels:
Date Opened:2004-02-09 10:38:38.000-0600Date Closed:2008-01-15 14:48:13.000-0600
Priority:BlockerRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:
I am running about 300 calls at the same time though pbx_spool.

I am getting these error messages.  Where you see the Call failed to go
though reason 5, the call is dropped and never tried again.  I looked at
the source code in ./pbx/pbx_spool.c and on line 199 I see the message but
I can't figure out what the error 5 is or how to fix it.

Feb  5 19:36:20 NOTICE[1335167792]: pbx_spool.c:199 attempt_thread: Call
failed to go through, reason 5

I also get


Feb  5 20:04:54 NOTICE[1588755248]: Call failed to go through, reason 1



****** ADDITIONAL INFORMATION ******

Feb  5 19:36:20 NOTICE[1343560496]: pbx_spool.c:199 attempt_thread: Call
failed to go through, reason 5
   -- Hungup 'IAX2[NuFone]/22'
   -- Call accepted by 66.225.202.72 (format GSM)
   -- Format for call is GSM
   -- Hungup 'IAX2[NuFone]/23'
Feb  5 19:36:20 NOTICE[1351953200]: pbx_spool.c:199 attempt_thread: Call
failed to go through, reason 5
   -- Call accepted by 66.225.202.72 (format GSM)
   -- Format for call is GSM
Feb  5 19:36:20 NOTICE[1360345904]: pbx_spool.c:199 attempt_thread: Call
failed to go through, reason 5
Comments:By: stefankroon (stefankroon) 2004-02-09 12:38:26.000-0600

Hi mlh,
I am not an asterisk expert, but a quick look to the code tells me that the reason you are getting is an asterisk control frame type. Code 5 is AST_CONTROL_BUSY and code 1 is AST_CONTROL_HANGUP. All the codes are described in the file include/asterisk/frame.h .
Hopefully this helps.
By: mlh (mlh) 2004-02-10 11:09:40.000-0600

Sorry not 300 calls  20 - 30 calls
By: Brian West (bkw918) 2004-02-10 14:40:19.000-0600

How many interfaces/lines/channels do you have?
By: mlh (mlh) 2004-02-14 17:03:05.000-0600

bkw: I have a t1 line and I am using NuFone for all out going calls
By: Brian West (bkw918) 2004-02-14 23:12:04.000-0600

How many call files are you trying to drop at once?  If call files exceed the outbound capacity then they will fail.
By: Mark Spencer (markster) 2004-02-15 11:57:17.000-0600

You can also use iax2 debug to see what is going on with the calls.
By: Mark Spencer (markster) 2004-02-23 00:17:36.000-0600

Can you confirm this is not simply running out of resources required for making your calls, either via iax2 debug or some other method?
By: zoa (zoa) 2004-02-23 11:47:43.000-0600

i think the spool retry thingie is fucked.

I was dumped 100 calls in there with a retry time of 10 minutes and 10 retry attempts.

I soon had 1000 call attempts at the same time and the box needed a power reset :)
By: James Golovich (jamesgolovich) 2004-03-08 14:13:32.000-0600

I know there are some other problems with the spool that I'm trying to dig into.  but one thing I came up with that helps with a lot of these dumping too many queue files into spool problem.  I've added a variable for the number of maximum outgoing calls generated by the spool at any one time.  Right now I have it just in pbx_spool.c, but if everyone thinks its a good idea I'll make it also check a config file (spool.conf or outgoing.conf) for this value.  I think a good default value would be 256, 512, or 1024.  On my test system I get all kinda of failures when its trying to generate more than about 200 calls at a time.  The limit will be able to be disabled by setting it to 0, so we can still do crazy stress testing

So if we think its a good idea I'll go ahead and implement the limit as a first step in dealing with this problem.
By: James Golovich (jamesgolovich) 2004-03-15 00:36:10.000-0600

Ok, I definitely see a problem with the code.  We are just simply processing the files multiple times.  I copied 2 files in right after each other and had 6 calls going on.  

Aha.  Think I got this one.  Got a fix, just doing some testing.  Seems we weren't really setting the mtime of the file when it was going to be retried.

I still think we should put a configurable hard limit on max queue events.  Any comments?
By: James Golovich (jamesgolovich) 2004-03-15 00:46:09.000-0600

Fixed in CVS.  Let me know if you still have problems.  and please give some opinions on a hard limit to outgoing call spool concurrent calls.  if you like the idea, then let me know how many calls you think a good default would be. 24, 48, 96, 128, 256, 512, 1024?
By: Malcolm Davenport (mdavenport) 2004-03-15 18:44:14.000-0600

Resolving pending further commentary.  Thanks, James.
By: zoa (zoa) 2004-03-16 03:25:46.000-0600

another problem i know of is that:

When you set a retry delay of 1 minute and your .call file will cause a call with a length of more than 1 minute (eg when calling to an answering machine, or maybe playing a song or whatever.) asterisk will keep retrying even if the call got answered succesfully.
By: James Golovich (jamesgolovich) 2004-03-20 00:40:35.000-0600

That is a likely scenario.  We have a quite a few potential ways to deal with it.  
1> A linked list containing active calls
2> Locking (with flock) and holding the lock of the file open until the call has completed
3> Renaming the file to include some kind of active flag
4> Adding some info to the file that the call is active, and remove it when inactive (doesnt seem like a very good idea to me)
By: Mark Spencer (markster) 2004-03-20 00:42:39.000-0600

I suggest qualifying the information (can't retry faster than retry) and that we log an entry to the file when it's answered, so that another retry can't happen.  Can you make it so, james?
By: Mark Spencer (markster) 2004-03-20 16:35:04.000-0600

Okay I've got a system that should be solid for not repeating while something is running but should handle a crash properly.
By: Digium Subversion (svnbot) 2008-01-15 14:48:13.000-0600

Repository: asterisk
Revision: 2506

U   branches/v1-0_stable/asterisk.c
U   branches/v1-0_stable/include/asterisk/options.h
U   branches/v1-0_stable/pbx/pbx_spool.c

------------------------------------------------------------------------
r2506 | markster | 2008-01-15 14:48:13 -0600 (Tue, 15 Jan 2008) | 3 lines

Correctly handle call flow with outgoing queue, avoiding retries while call acti
ve (bug ASTERISK-1012)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=2506
By: Digium Subversion (svnbot) 2008-01-15 14:48:13.000-0600

Repository: asterisk
Revision: 2505

U   trunk/asterisk.c
U   trunk/include/asterisk/options.h
U   trunk/pbx/pbx_spool.c

------------------------------------------------------------------------
r2505 | markster | 2008-01-15 14:48:12 -0600 (Tue, 15 Jan 2008) | 3 lines

Correctly handle call flow with outgoing queue, avoiding retries while call acti
ve (bug ASTERISK-1012)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=2505