Summary:	ASTERISK-30447: Stasis/p:channel:all reaching 500 tasks
Reporter:	Luke Escude (lukeescude)	Labels:
Date Opened:	2023-03-01 11:01:10.000-0600	Date Closed:	2023-04-11 15:49:45
Priority:	Minor	Regression?
Status:	Closed/Complete	Components:	Applications/app_stasis
Versions:	16.30.0	Frequency of Occurrence
Related Issues:
Environment:		Attachments:
Description:	Starting last week, some of our larger Asterisk instaces are logging a bunch of these: The 'stasis/p:channel:all-0000513d' task processor queue reached 500 scheduled tasks. I haven't found much about the p:channel:all task processor (there's some documentation around the others, but not this one).
Comments:	By: Asterisk Team (asteriskteam) 2023-03-01 11:01:13.451-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Joshua C. Colp (jcolp) 2023-03-01 11:06:19.846-0600 Asterisk 16 no longer receives bug fixes. As for that topic, it is a topic for all events for all channels. The more channels, the more events. Depending on usage it can be subscribed to by AMI, so if an AMI consumer is slow that may have an impact. 16 may have stasis statistics if you enable developer mode, they show up under the "stasis statistics" CLI commands which can narrow down what is backed up in stasis. As I originally stated though no changes would be done to Asterisk 16. By: Luke Escude (lukeescude) 2023-03-01 11:16:26.642-0600 Okay that helps - I have two possible culprits: 1. Webhooks - these particular customers have POST webhooks occuring throughout their channels, as we send real-time call events to their 3rd party API. Since the CURL functions are blocking, a 2-second timeout being reached could be backing up the channel. 2. AMI monitoring daemon - As AMI feeds events to the monitoring system, it's possible I should queue up the events in a queue, instead of submitting them to our redis cluster in real-time, so it can free up the AMI socket more quickly. Let me know if my logic doesn't make sense, and also I will look at upgrading to the next LTS version - Thanks Joshua! By: Joshua C. Colp (jcolp) 2023-03-01 11:19:46.577-0600 It's entirely possible those are the causes, but that's all I can say. By: Luke Escude (lukeescude) 2023-03-01 11:21:45.476-0600 Perfect I'll implement those fixes, and will report back here with a result in case someone else ever ends up with a similar issue. By: Luke Escude (lukeescude) 2023-03-08 16:16:43.050-0600 Well, neither of those helped, although the performance improvement in dial plan processing is noticeable. We're having that reseller move some tenants off the overloaded system, so I think Asterisk just can't handle what we're doing to it. It's a pretty high-volume system: 20+ active calls at any given time, 450+ extensions, almost 3,000 hints registered. So we can close this until I accidentally stumble on a solution. By: Luke Escude (lukeescude) 2023-04-11 15:24:57.543-0500 Hey Joshua, So this began happening the day I disabled Asterisk CDR. Re-enabling CDR seems to fix it. A few months ago I set enabled=no in cdr.conf, since we're not using CDR. By: Asterisk Team (asteriskteam) 2023-04-11 15:24:57.856-0500 This issue has been reopened as a result of your commenting on it as the reporter. It will be triaged once again as applicable. By: Joshua C. Colp (jcolp) 2023-04-11 15:39:31.690-0500 Ok. If it can be reproduced in 18 or above then this could remain open. By: Luke Escude (lukeescude) 2023-04-11 15:46:49.456-0500 Sounds good, you can suspend/close this until I get our system upgraded to 20.