[Home]

Summary:ASTERISK-14183: Error rate for 'asterisk.c: Accept returned -1: Too many open files' is not throttled.
Reporter:Ernie Dunbar (ernied)Labels:
Date Opened:2009-05-21 11:30:28Date Closed:2011-06-07 14:01:00
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:When Asterisk 1.4.23.1 runs out of available filehandles, the log file grows out of control and fills the disk in a very short period of time. When this error occurs, approximately 700-800 lines of logging are produced each second. This jams the system and essentially crashes Asterisk. Unless Asterisk crashes first as a result of this error message that is, but the two events happen so closely together that it's impossible to tell.



****** ADDITIONAL INFORMATION ******

I'm not entirely sure where the error comes from, but it's worth noting that we've set up a configuration scheme where each SIP user has their own configuration file, and the entire directory is read with the directive #include /etc/asterisk/sipdir/*. The same goes for dialplans. There are around 1400 files in total. I have increased the ulimit to 32768 compensate for this, but it still happens. Also, it takes several days to a week of continuous uptime before the problem occurs.
Comments:By: Tilghman Lesher (tilghman) 2009-05-21 12:47:32

The proper response to this is not to throttle the error message, but to figure out where the file descriptors are leaking.  Please 'make menuselect', select compiler options and enable DEBUG_FD_LEAKS.  After 24 hours of running with DEBUG_FD_LEAKS, please run the CLI command "core show fd" and upload the output into the file upload area of this issue.
By: Ernie Dunbar (ernied) 2009-05-21 13:01:48

There doesn't appear to be a "Compiler Options" in the make menuconfig menu. Selecting "Compiler Flags" results in these options:

1.  DONT_OPTIMIZE
2.  DEBUG_CHANNEL_LOCKS
3.  DEBUG_THREADS
4.  LOW_MEMORY
5.  MALLOC_DEBUG
6.  RADIO_RELAX
7.  STATIC_BUILD
8.  IAX_OLD_FIND
9.  LOADABLE_MODULES

Perhaps I need to use a manual compiler flag?
By: Tilghman Lesher (tilghman) 2009-05-21 14:10:39

No, you just need to upgrade.
By: Ernie Dunbar (ernied) 2009-05-21 14:36:24

To what? 1.6? Not going to happen anytime soon.

1.4.24.1 doesn't have those compiler flags either.
By: Tilghman Lesher (tilghman) 2009-05-21 15:28:11

1.4 SVN has that compiler flag, and it's also in 1.4.25-rc1.
By: Tilghman Lesher (tilghman) 2009-05-21 15:58:53

And as of an hour ago, 1.4.25 has been released.  Note that this release may in fact already address your problems with leaking file descriptors, as part of the reason it was written was to diagnose leaking file descriptors in 1.4.
By: Ernie Dunbar (ernied) 2009-05-21 18:33:44

Ok, I'll upgrade to that tonight.
By: Ernie Dunbar (ernied) 2009-05-26 12:14:58

Uh, version 1.4.24 and up are horribly unstable on our Gentoo system using kernel 2.6.27-gentoo-r7. We honestly can't expect more than 30 minutes of uptime.
By: Tilghman Lesher (tilghman) 2009-05-26 12:39:12

I'm not sure what you mean by 'horribly unstable'.  Is it crashing?  If so, can you give us a backtrace?
By: Ernie Dunbar (ernied) 2009-05-26 13:15:36

Oh yes, we get segfaults like this:

May 26 10:17:58 voip asterisk[25727]: segfault at 3046 ip 00003046 sp b79ba87c error 4 in asterisk[8048000+103000]

Asterisk was crashing all morning at random intervals no more than 30 minutes apart. I'll try to see if I can reproduce the problem on our backup server and get a backtrace.
By: Leif Madsen (lmadsen) 2009-07-13 10:06:54

Closed due to lack of response. Please reopen if you can provide the requested information. Thanks!