[Home]

Summary:ASTERISK-03141: [patch] Enhancements to asterisk.c:set_priority: Get RT without runaway risk?
Reporter:stevekstevek (stevekstevek)Labels:
Date Opened:2004-12-30 16:10:09.000-0600Date Closed:2011-06-07 14:04:42
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) asterisk-dropprio-fork.diff.txt
( 1) patch-freebsd-rtprio-asterisk.c
Description:Presently, you can run asterisk with the "-p" option, to set it to use realtime scheduling on linux, or negative niceness on other unices.

I'd like to see this enhanced, in the following ways:

1) Set RealTime priority on other operating systems (Mac OS X, *BSD, etc), where available.

2) Drop realtime priority when forking external processes (AGI, etc):  You definately don't need real-time for these, and you don't need a perk interpreter starting up with real-time priority.

3) Prevent runaways.  Ideally, there would be a way to tell the operating system "I'd like this process to run with real-time priority for up to 90% of the CPU bandwidth, then get regular scheduling priority for the rest".   This would be ideal, as it would give 100% to asterisk if nothing else was going on, but it would ensure the rest of the system would still get 10% in even if asterisk was very busy.

At the very least, we should have some system in place to prevent asterisk from running away in a spin-loop on your box, and making it unusable.  I've implemented this in iaxclient:

http://cvs.sourceforge.net/viewcvs.py/iaxclient/iaxclient/lib/unixfuncs.c?rev=1.4&view=markup

based on work in portaudio;  Basically, this sets up two threads:  one sched_other thread, and one higher-priority realtime thread.  Once per second, the threads need to "talk" to each other.  If they can't for 4 seconds, the higher-priority thread lowers the "monitored" application to normal priority.

What this means is that if you do for(;;) { getpid() ;} in your real-time thread, your machine only locks up for 4 seconds, instead of forever.

This isn't ideal for asterisk, unless the real-timedness were to be re-conferred after some period of time, because the CPU-hogging could just be actual high-loads..

Mac OS X has a nice API for doing what I'd like, but it's actually implemented more like iaxclient/linux is, where it just gives the process real-time priority, and then takes away it's real-timedness if it hogs the machine for more than a few seconds at a time..

See:
http://lists.apple.com/archives/darwin-development/2004/Feb/msg00079.html

The API is like this:
     /* give us at least how much? 6-8ms every 10ms (we generally need less) */
     ttcpolicy.period = 10 * hzms; /* 10 ms */
     ttcpolicy.computation = 2 * hzms;
     ttcpolicy.constraint = 3 * hzms;
     ttcpolicy.preemptible = 1;

     if ((ret=thread_policy_set(mach_thread_self(),
       THREAD_TIME_CONSTRAINT_POLICY, (int *)&ttcpolicy,
       THREAD_TIME_CONSTRAINT_POLICY_COUNT)) != KERN_SUCCESS) {
           fprintf(stderr, "thread_policy_set failed: %d.\n", ret);
     }

It would be awesome if there were some way to specify a policy like that under linux (maybe there's some watchdog wizardry that can do this from userspace, with the existing APIs?)
Comments:By: petersv (petersv) 2005-01-01 06:06:54.000-0600

We have implemented the feature 2) above. Our legal department is sitting on the disclaimer over the holidays, otherwise we would have submitted it already. Unless someone beats us to it we will submit it.
By: stevekstevek (stevekstevek) 2005-01-05 12:11:21.000-0600

petersv, please put the bug number for your patch here, once you have it ready..
By: stevekstevek (stevekstevek) 2005-01-20 17:49:40.000-0600

petersv:  Got that patch ready?

I'll write (3), if you have (2)..
By: stevekstevek (stevekstevek) 2005-01-20 17:51:22.000-0600

Reminder sent to stevekstevek

reminding myself of something
By: petersv (petersv) 2005-01-21 02:00:29.000-0600

stevekstevek: Sorry, the legal department is still sitting on the disclaimer. I can send you the patch by email if you want to review it. Email me at "psvasterisk at psv dot nu".
By: Mark Spencer (markster) 2005-02-19 15:25:38.000-0600

Any update here?
By: petersv (petersv) 2005-02-20 13:47:28.000-0600

I've been ill for a few weeks and without me pushing for the disclaimer to be accepted the legal department seems to have put it on the bottom of the pile. I'll attach the patch for comments. I'll add a note as soon as the disclaimer has been signed.
By: Michael Jerris (mikej) 2005-03-09 12:18:48.000-0600

Reminder sent to petersv

Any word on this?
By: petersv (petersv) 2005-03-09 14:04:29.000-0600

Hopefully after Cebit.
By: stevekstevek (stevekstevek) 2005-03-09 16:15:43.000-0600

Petersv's patch doesn't actually solve the issue I'm trying to address here, but is definitely a good idea.

If he can't get a sign-off on this, I can send in a patch for that as well, although I would have just called the function

int ast_fork(int dropprio), and had it drop RT Prio iff dropprio was non-zero;  I'd also have it drop nice to zero if it was set below zero.  This should even improve things when using, e.g. AGI, and starting asterisk with nice -10 asterisk, etc.

I'll also have someone here write/test the Real-Time watchdog process thing within a few weeks, but if the mantis-masters want, I can submit a patch for that separately (although, it can be done as a separate program).
By: petersv (petersv) 2005-03-09 17:27:45.000-0600

stevekstevek: Go ahead, it will be faster that way. After cebit (as well as for over a month before) the legal department is tied up preparing / closing a lot of deals.

I'd recommend against dropping nice-ness. Being nice does not create a lock-out situation. Most things asterisk does (such as MoH and running agi:s) should take precedence over other operations on the machine. The do not need the absolute priority of a RT scheduler.
By: staffanu (staffanu) 2005-03-25 15:18:44.000-0600

I've implemented 1), i.e., using rtprio on FreeBSD.  I'll upload a patch (which is against CVS per 2005-03-22) in a few minutes.
By: stevekstevek (stevekstevek) 2005-03-26 17:02:13.000-0600

I'm going to write a patch (maybe next week), which does the following:

1) Includes the basic idea from petersv, creating an "ast_fork(int dropRT)", which will fork, and optionally drop RT priority in the child.

2) Adds a new program (this might actually go in asterisk-addons or something), which starts asterisk with real-time priority N, and starts two other processes, one with normal priority, and one with RT Priority N+1.  The normal priority process will yelp to the N+1 priority process occassionally, and if the N+1 priority thread doesn't hear from it in a couple of intervals, it will take away the real-time priority from asterisk, assuming it is in some kind of deadly spinloop or something.

I've found that with iaxclient, I get better audio performance on linux when using Real-Time, but I suspect that there's going to be problems with asterisk using it Real-Time, because there's no real scheduling at all that happens for real-time processes; each thread will run until it blocks, and the threads at the same priority will not preempt each other.

This means, that if you do CLI reload, for example, that thread is going to stop all audio processing until it's done, or it blocks.  "show translation recalc" will also probably be bad :)

All this probably matters most for people who for whatever reason can't avoid running asterisk on a box with other stuff running on it..
By: petersv (petersv) 2005-03-26 17:32:36.000-0600

Threads with the realtime scheduler will actually preempt one another if they are SCHED_RR. SCHED_FIFO will not. See "man sched_setscheduler". Threads with the same prio and the SCHED_RR scheduler will behave the same way they do under SCHED_OTHER.
By: stevekstevek (stevekstevek) 2005-04-07 10:33:55

We're working on this now..

But, I just found this, which would actually be much better, if it were in kernels..:

http://people.redhat.com/mingo/rt-limit-patches/
By: Michael Jerris (mikej) 2005-05-23 21:42:43

Is this one still alive?
By: staffanu (staffanu) 2005-05-24 16:36:17

I believe the freebsd-rtprio patch could be applied independently of resolving the, obviously more difficult, other issues.
By: stevekstevek (stevekstevek) 2005-05-24 17:53:19

I have two patches that should be coming soon:

1) A patch similar to the dropprio-fork patch above, which creates a single new API function ast_fork(int dropprio), which is just like fork, but may drop the child process down to normal priority.

2) A watchdog/wrapper program, which spawns a watchdog and a canary, and then spawns an asterisk process with real-time privileges (optionally changing uid as well).  

We've been using these for a couple of weeks now, and just need to tidy them up before we post them.  (2) above, may be something that can live in asterisk-contrib or something -- although it is called "asterisk-rt", it's actually generic enough to start _any_ program with real-time priority.
By: knielsen (knielsen) 2005-05-25 04:40:21

It would be great if the watchdog could run at priority N+10 (say), not just N+1. Then individual threads could bump themselves up if they need priority over the rest of Asterisk. For example, I would run the CLI threads and monitor threads waiting for calls at N, threads doing realtime audio stuff (bridging...) at N+1 (to help audio glitches), and an MTP2 thread at N+2 (to avoid frame loss taking the signalling link down).
By: Clod Patry (junky) 2005-06-08 07:21:55

stevekstevek: on 05-24-05, you said you will post patches soon. Do you have anything to publish at the moment?
By: Michael Jerris (mikej) 2005-06-20 21:47:12

Suspended due to lack of response.  Stevek-  Please re-open when you have patches for this ready.