[Home]

Summary:ASTERISK-07319: [patch] Asterisk crashes when rtcp struct fails allocation due to too many files open
Reporter:jmls (jmls)Labels:
Date Opened:2006-07-10 12:02:30Date Closed:2006-07-31 10:25:40
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/RTP
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) corebt-1.txt
( 1) corebt-2.txt
( 2) corebt-3.txt
( 3) corebt-4.txt
( 4) rtp.c.37320.patch
Description:We are running svn trunk r37320, and have had several segfaults today. I have attached 4 bt fulls on the core dumps, and they all seem to point to line 1819 of rtp.c.

Comments:By: John Martin (jfp_martin) 2006-07-11 04:13:24

jmls,
 I've had a look at your core bt's and the code and it's possible that a seg fault is being caused by rtcp failing and not being accounted for. I'm not sure why rtcp might have failed but the patch I submitted may fix the seg fault, though not the cause of the rtcp error.

 Do you have any errors in your asterisk log file regarding rtp or rtcp?
 Do you have a limited number of ports assigned to rtp in rtp.conf?
 Do you run any rtp debug during the calls you made?

John
By: jmls (jmls) 2006-07-11 05:14:14

Thanks for the patch. I will apply it tonight and report back.

In response to your questions.
1) these are the errors regarding rtp

Jul 10 08:48:21 ERROR[17718] rtp.c: Unable to allocate socket: Too many open files
Jul 10 08:48:21 WARNING[17718] chan_sip.c: Unable to create RTP audio  session: Too many open files

(asterisk crashed at 08:48)

Jul 10 09:11:29 ERROR[24865] rtp.c: Unable to allocate socket: Too many open files
Jul 10 09:11:29 WARNING[24865] chan_sip.c: Unable to create RTP audio  session: Too many open files
Jul 10 09:11:40 ERROR[408] rtp.c: Unable to allocate socket: Too many open files
Jul 10 09:11:40 WARNING[408] chan_sip.c: Unable to create RTP audio  session: Resource temporarily unavailable

(asterisk did not crash at 09:11)

Jul 10 09:26:29 ERROR[24865] rtp.c: Unable to allocate socket: Too many open files
Jul 10 09:26:29 WARNING[24865] chan_sip.c: Unable to create RTP audio  session: Too many open files

(asterisk did not crash at 09:26)

Jul 10 09:30:37 WARNING[24888] rtp.c: Unable to allocate RTCP socket: Too many open files
Jul 10 09:30:37 WARNING[5787] rtp.c: Unable to allocate RTCP socket: Too many open files

(asterisk crashed at 09:30)

Jul 10 09:55:07 WARNING[6469] rtp.c: Unable to allocate RTCP socket: Too many open files

(asterisk crashed at 09:30)

Jul 10 12:19:46 WARNING[7623] rtp.c: Unable to allocate RTCP socket: Too many open files

(asterisk crashed at 12:19)

Jul 10 13:20:57 WARNING[12711] rtp.c: Unable to allocate RTCP socket: Too many open files
Jul 10 13:20:58 ERROR[17424] rtp.c: Unable to allocate socket: Too many open files
Jul 10 13:20:58 WARNING[17424] chan_sip.c: Unable to create RTP audio  session: Too many open files
Jul 10 13:21:01 WARNING[12688] rtp.c: Unable to allocate RTCP socket: Too many open files

(asterisk crashed at 13:21)

Jul 10 15:53:47 ERROR[21433] rtp.c: RTCP RR transmission error to, rtcp halted No such file or directory

(asterisk did not crash at 15:53)

So, it looks as if you are right - the system had too many open files, caused rtp to error, causing the segfault.

Interestingly enough I fixed a typo in the dialplan at 13:30 (For a ringall queue I had added local members instead of SIP members) and didn't have a crash since then.
By: jmls (jmls) 2006-07-11 05:15:06

point 2) I used the standard rtp.conf file

; RTP Configuration
;
[general]
;
; RTP start and RTP end configure start and end addresses
;
; Defaults are rtpstart=5000 and rtpend=31000
;
rtpstart=10000
rtpend=20000
;
; Whether to enable or disable UDP checksums on RTP traffic
;
;rtpchecksums=no
;
; The amount of time a DTMF digit with no 'end' marker should be
; allowed to continue (in 'samples', 1/8000 of a second)
;
;dtmftimeout=3000

Should I increase the number of ports ?
By: jmls (jmls) 2006-07-11 05:15:32

Point 3) Sorry, I wasn't running rtp debug at the time.
By: John Martin (jfp_martin) 2006-07-11 06:06:27

jmls,
1)  looks like my theory was right, though it will be interesting to see what results you get when you apply the patch. The patch won't fix you running out of files, but it should stop the seg fault. I've seen issues with people not having enough headroom on opening files, you should be able to get google to come up with a solution to that.
The fact you were using local channels for the queue _shouldn't_ cause a problem if you have enough file descriptors available... but, it will use more file descriptors as the calls exit the queue and so removing the local channel will give you more breathing space. You may find now that as you increase the number of active calls you'll get to a point where you start to run out of files again.

2) you don't need to increase the number of ports in rtp.conf. It was just one of the failure mechanisms I could see. 10000 ports should be plenty.

3) again, having debug on may have caused some changes in timing and I was trying to narrow down a cause. If you had debug on then a) it might have given us a clue  but b) turning it off may have told us something else :-)

I would recommend you look to increase the number of files available on your system and please let me (us) know how you get on.

John
By: Serge Vecher (serge-v) 2006-07-11 08:44:19

Updated description.
AuPix: can you please confirm you Disclaimer status? Thanks for the patch
By: John Martin (jfp_martin) 2006-07-11 08:46:52

Vechers,
 I confirm that I have a disclaimer filed and that this patch is covered by it.

John
By: jmls (jmls) 2006-07-11 17:19:39

I have applied the patch. I will let you know what happens. Many thanks.
By: John Martin (jfp_martin) 2006-07-21 08:49:04

jmls,
 any news on the workaround. Without feedback the bug marshals won't put the patch into SVN (unless they're feeling generous :-)

John
By: jmls (jmls) 2006-07-30 17:22:47

Sorry, have been on holiday for the past two weeks - however, I did apply the patch, and * has died twice in the last two weeks, something to do with sip transfers.

So, I can say that this patch certainly has made thing much much better.

Thanks.
By: John Martin (jfp_martin) 2006-07-31 03:21:24

Thanks jmls.

Vechers can this be put in ASAP. This bug could cause all sorts of other problems in 1.4 so would be advisable to get it in before the beta phase - IMHO.

John
By: Joshua C. Colp (jcolp) 2006-07-31 10:25:39

Fixed in trunk as of revision 38573. This only covers the attached patch for making sure an RTCP structure exists before trying to muck with it. As for running out of file descriptors - bad things will happen...