[Home]

Summary:ASTERISK-05734: BYE instead of response causes stuck channel
Reporter:Alistair Cunningham (acunningham)Labels:
Date Opened:2005-11-29 08:10:32.000-0600Date Closed:2011-06-07 14:10:49
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) debug
( 1) tmp3.txt
( 2) tmp4.txt
Description:When Asterisk calls another system with the following SIP packets:

Asterisk -> Other system: INVITE
Other system -> Asterisk: 100 Trying
Other system -> Asterisk: BYE
Asterisk -> Other system: 200 OK (CSeq is for the BYE)

Asterisk creates a channel which never gets cleaned up. A 'sip show channels' shows many entries like:

1.2.3.4   1234567890  1234567890  00102/00001  unkn  No  (d)  Rx: BYE

These gradually get more and more, and can be hundreds. We suspect that eventually they cause Asterisk instability, but have not been able to prove this.

This occurs on both 1.2.0 and CVS HEAD as of 1st November 2005. It could be related to bug ASTERISK-334.

Traces are available on request to Digium staff, but our customer has requested that they not be made public to keep IP address and numbers private.
Comments:By: Serge Vecher (serge-v) 2005-11-29 17:05:20.000-0600

acunningham: just substitute IP addresses/passwords with different values and post the logs here (as attachments, please).
By: Alistair Cunningham (acunningham) 2005-11-30 07:06:23.000-0600

I've uploaded the trace, replacing the following:

Calling number: 1234567890
Called number: 9876543210
Asterisk IP address: 1.1.1.1
Remote system: 1.2.3.4
By: Alistair Cunningham (acunningham) 2005-11-30 07:07:46.000-0600

The corresponding line in "sip show channels" is:

1.2.3.4   1234567890  18cc16a5662  00102/00001  unkn  No  (d)
Rx: BYE
By: Olle Johansson (oej) 2005-12-01 16:29:05.000-0600

For SIP debug output, we always need you to turn debug to level 4 and verbose to level 4. Thank you.
By: Olle Johansson (oej) 2005-12-01 16:35:56.000-0600

From RFC3261, section 15:
"A UA MUST NOT send a BYE outside of
a dialog. The caller’s UA MAY send a BYE for either confirmed or early dialogs, and the callee’s UA MAY
send a BYE on confirmed dialogs, but MUST NOT send a BYE on early dialogs."

The user agent seems to be non-conformant to the RFC.
By: Olle Johansson (oej) 2005-12-01 16:36:32.000-0600

...this is an early dialog and the UA should send a failure response to the INVITE, not a BYE.
By: Alistair Cunningham (acunningham) 2005-12-02 05:48:59.000-0600

I agree that the far end breaks the RFC, and we've asked the owner of that system to contact their vendor to get the problem fixed. It's entirely proper for Asterisk to reject this message.

What we feel is not correct for Asterisk to do is to keep track of the BYE message for a long time, with an ever growing number of channels. We feel it should discard these channels either immediately, or after a short timeout.

If nothing else, Asterisk might well be vulnerable to a denial of service attack through this.

I'll try to get a trace with 'set debug 4' and 'set verbose 4' today, and will upload it.
By: Alistair Cunningham (acunningham) 2005-12-02 07:22:04.000-0600

I've attached a debug file. It's been edited to remove confidential information and other calls.

The verbose output is identical to the file already uploaded (tmp3.txt), which was taken with verbose 9.
By: Olle Johansson (oej) 2005-12-02 10:14:29.000-0600

Sorry, while pointing out that the remote end is doing the wrong thing, I completely agree that we should not keep open channels. I've been looking into the code trying to find this...
By: Olle Johansson (oej) 2005-12-02 10:15:55.000-0600

First make sure that both debug and verbose are logged to the console, you have no verbose output in tmp3.txt.

I need one combined file with SIP debug output, and level 4 verbose and debug logging. Thank you.
By: Alistair Cunningham (acunningham) 2005-12-03 10:41:09.000-0600

tmp4.txt contains the console output with 'set debug 4', 'set verbose 4', 'sip debug ip 1.2.3.4', and 'console => notice,warning,error,debug' in logger.conf. There's no verbose output as none occurred during the outbound call.
By: Olle Johansson (oej) 2005-12-03 12:03:44.000-0600

Please continue to save logging after the BYE and do a "sip show channels". Turn on SIP history and dumphistory in sip.conf. Just wait and watch for any destroyed channels. Do a "SIP show history" on a channel that stays alive even though it should not. Thank you.
By: Alistair Cunningham (acunningham) 2005-12-03 12:15:17.000-0600

This is going to be difficult, as there is a lot of other traffic on the system (getting the trace in tmp4.txt was not easy), and any trace I upload needs to be edited to remove all real IP addresses and numbers.

Is it really necessary? The scenario is well defined, and we know exactly the packets that cause it.

How about I set up a test server which makes calls to another test Asterisk server then inject BYEs using sipsak?
By: Olle Johansson (oej) 2005-12-03 13:35:37.000-0600

Whatever, The interesting part begins where you stop logging - after the bye. Something prevents the call from being destroyed.
By: Alistair Cunningham (acunningham) 2005-12-03 18:12:44.000-0600

What should I be looking for after the BYE? I've been doing some experimentation, and uploading the whole trace is not going to be feasible as there's a large volume of other traffic, and the customer doesn't want IP addresses or telephone numbers made public.
By: Alistair Cunningham (acunningham) 2005-12-06 10:33:04.000-0600

We're now sure that this can cause Asterisk to block console connections by running Asterisk out of file descriptors, even when these are set to 65535.

/var/log/asterisk/messages gives:

Dec  6 18:11:12 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files
Dec  6 18:11:17 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files
Dec  6 18:11:25 ERROR[3304] asterisk.c: Unable to create pipe: Too many open files

# su - asterisk -c 'ulimit -a'
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 65535
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) unlimited
virtual memory        (kbytes, -v) unlimited

At this time, 'sip show channels' showed 497 SIP channels, 4 of which were real, and 493 were due to the BYEs.
By: Olle Johansson (oej) 2005-12-06 13:26:57.000-0600

Anyway I can get access to the system?

Or, if you trust me handling the information, mail me a debug file.

I'm having problems repeating this. And yes, stuck SiP channels will cause us hitting the roof on file descriptors.
By: Alistair Cunningham (acunningham) 2005-12-06 15:04:55.000-0600

oej,

The customer has agreed to give you remote access subject to an NDA. Can you please give me your email address? I'll then send you the NDA.
By: Alistair Cunningham (acunningham) 2005-12-08 04:53:11.000-0600

oej,

Bad news, I'm afraid. I'm told that the equipment sending the BYEs has now been replaced, so we can't take traces. I've tried to find out what sort of device it was, and was told it was made by Voipswitch, but we don't know the model.

I have no objections to you closing this ticket, or leaving it open to continue working on it.

I'm sorry that we haven't been able to help solve this. My apologies for this.
By: Olle Johansson (oej) 2006-01-04 14:36:07.000-0600

Closing, since this obviously is no problem any more. Need to keep this in mind as a possible concern in the future. Thanks for responding quickly and wanting to provide access and requested log files.

/Olle