Summary: | ASTERISK-05734: BYE instead of response causes stuck channel | ||
Reporter: | Alistair Cunningham (acunningham) | Labels: | |
Date Opened: | 2005-11-29 08:10:32.000-0600 | Date Closed: | 2011-06-07 14:10:49 |
Priority: | Minor | Regression? | No |
Status: | Closed/Complete | Components: | Core/General |
Versions: | Frequency of Occurrence | ||
Related Issues: | |||
Environment: | Attachments: | ( 0) debug ( 1) tmp3.txt ( 2) tmp4.txt | |
Description: | When Asterisk calls another system with the following SIP packets: Asterisk -> Other system: INVITE Other system -> Asterisk: 100 Trying Other system -> Asterisk: BYE Asterisk -> Other system: 200 OK (CSeq is for the BYE) Asterisk creates a channel which never gets cleaned up. A 'sip show channels' shows many entries like: 1.2.3.4 1234567890 1234567890 00102/00001 unkn No (d) Rx: BYE These gradually get more and more, and can be hundreds. We suspect that eventually they cause Asterisk instability, but have not been able to prove this. This occurs on both 1.2.0 and CVS HEAD as of 1st November 2005. It could be related to bug ASTERISK-334. Traces are available on request to Digium staff, but our customer has requested that they not be made public to keep IP address and numbers private. | ||
Comments: | By: Serge Vecher (serge-v) 2005-11-29 17:05:20.000-0600 acunningham: just substitute IP addresses/passwords with different values and post the logs here (as attachments, please). By: Alistair Cunningham (acunningham) 2005-11-30 07:06:23.000-0600 I've uploaded the trace, replacing the following: Calling number: 1234567890 Called number: 9876543210 Asterisk IP address: 1.1.1.1 Remote system: 1.2.3.4 By: Alistair Cunningham (acunningham) 2005-11-30 07:07:46.000-0600 The corresponding line in "sip show channels" is: 1.2.3.4 1234567890 18cc16a5662 00102/00001 unkn No (d) Rx: BYE By: Olle Johansson (oej) 2005-12-01 16:29:05.000-0600 For SIP debug output, we always need you to turn debug to level 4 and verbose to level 4. Thank you. By: Olle Johansson (oej) 2005-12-01 16:35:56.000-0600 From RFC3261, section 15: "A UA MUST NOT send a BYE outside of a dialog. The caller’s UA MAY send a BYE for either confirmed or early dialogs, and the callee’s UA MAY send a BYE on confirmed dialogs, but MUST NOT send a BYE on early dialogs." The user agent seems to be non-conformant to the RFC. By: Olle Johansson (oej) 2005-12-01 16:36:32.000-0600 ...this is an early dialog and the UA should send a failure response to the INVITE, not a BYE. By: Alistair Cunningham (acunningham) 2005-12-02 05:48:59.000-0600 I agree that the far end breaks the RFC, and we've asked the owner of that system to contact their vendor to get the problem fixed. It's entirely proper for Asterisk to reject this message. What we feel is not correct for Asterisk to do is to keep track of the BYE message for a long time, with an ever growing number of channels. We feel it should discard these channels either immediately, or after a short timeout. If nothing else, Asterisk might well be vulnerable to a denial of service attack through this. I'll try to get a trace with 'set debug 4' and 'set verbose 4' today, and will upload it. By: Alistair Cunningham (acunningham) 2005-12-02 07:22:04.000-0600 I've attached a debug file. It's been edited to remove confidential information and other calls. The verbose output is identical to the file already uploaded (tmp3.txt), which was taken with verbose 9. By: Olle Johansson (oej) 2005-12-02 10:14:29.000-0600 Sorry, while pointing out that the remote end is doing the wrong thing, I completely agree that we should not keep open channels. I've been looking into the code trying to find this... By: Olle Johansson (oej) 2005-12-02 10:15:55.000-0600 First make sure that both debug and verbose are logged to the console, you have no verbose output in tmp3.txt. I need one combined file with SIP debug output, and level 4 verbose and debug logging. Thank you. By: Alistair Cunningham (acunningham) 2005-12-03 10:41:09.000-0600 tmp4.txt contains the console output with 'set debug 4', 'set verbose 4', 'sip debug ip 1.2.3.4', and 'console => notice,warning,error,debug' in logger.conf. There's no verbose output as none occurred during the outbound call. By: Olle Johansson (oej) 2005-12-03 12:03:44.000-0600 Please continue to save logging after the BYE and do a "sip show channels". Turn on SIP history and dumphistory in sip.conf. Just wait and watch for any destroyed channels. Do a "SIP show history" on a channel that stays alive even though it should not. Thank you. By: Alistair Cunningham (acunningham) 2005-12-03 12:15:17.000-0600 This is going to be difficult, as there is a lot of other traffic on the system (getting the trace in tmp4.txt was not easy), and any trace I upload needs to be edited to remove all real IP addresses and numbers. Is it really necessary? The scenario is well defined, and we know exactly the packets that cause it. How about I set up a test server which makes calls to another test Asterisk server then inject BYEs using sipsak? By: Olle Johansson (oej) 2005-12-03 13:35:37.000-0600 Whatever, The interesting part begins where you stop logging - after the bye. Something prevents the call from being destroyed. By: Alistair Cunningham (acunningham) 2005-12-03 18:12:44.000-0600 What should I be looking for after the BYE? I've been doing some experimentation, and uploading the whole trace is not going to be feasible as there's a large volume of other traffic, and the customer doesn't want IP addresses or telephone numbers made public. By: Alistair Cunningham (acunningham) 2005-12-06 10:33:04.000-0600 We're now sure that this can cause Asterisk to block console connections by running Asterisk out of file descriptors, even when these are set to 65535. /var/log/asterisk/messages gives: Dec 6 18:11:12 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files Dec 6 18:11:17 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files Dec 6 18:11:25 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files # su - asterisk -c 'ulimit -a' core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited At this time, 'sip show channels' showed 497 SIP channels, 4 of which were real, and 493 were due to the BYEs. By: Olle Johansson (oej) 2005-12-06 13:26:57.000-0600 Anyway I can get access to the system? Or, if you trust me handling the information, mail me a debug file. I'm having problems repeating this. And yes, stuck SiP channels will cause us hitting the roof on file descriptors. By: Alistair Cunningham (acunningham) 2005-12-06 15:04:55.000-0600 oej, The customer has agreed to give you remote access subject to an NDA. Can you please give me your email address? I'll then send you the NDA. By: Alistair Cunningham (acunningham) 2005-12-08 04:53:11.000-0600 oej, Bad news, I'm afraid. I'm told that the equipment sending the BYEs has now been replaced, so we can't take traces. I've tried to find out what sort of device it was, and was told it was made by Voipswitch, but we don't know the model. I have no objections to you closing this ticket, or leaving it open to continue working on it. I'm sorry that we haven't been able to help solve this. My apologies for this. By: Olle Johansson (oej) 2006-01-04 14:36:07.000-0600 Closing, since this obviously is no problem any more. Need to keep this in mind as a possible concern in the future. Thanks for responding quickly and wanting to provide access and requested log files. /Olle |