Summary:	ASTERISK-05828: Asterisk blocks when a long call causes Monitor() to generate a 2GB file.
Reporter:	Nick Barnes (bcnit)	Labels:
Date Opened:	2005-12-13 06:30:11.000-0600	Date Closed:	2006-02-20 16:36:41.000-0600
Priority:	Minor	Regression?	No
Status:	Closed/Complete	Components:	Core/General
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) 20051219__initial64bit_on_32bit.diff.txt ( 1) 20060220__bug5984.diff.txt
Description:	NOTE - This bug was found on a live 1.0.9 installation. I have been unable to test this on a 1.2 installation. Sorry for listing this under 1.2, but I can't find any trace of this bug ever being reported and thought that it may never have been fixed and will therefore still be in 1.2. Sorry again. When Monitor() is used to record a call which lasts long enough to create a 2GB file, Asterisk: 1 - Creates lots and lots of event_log.XXXXX and messages.XXXXX files 2 - Blocks The process stays up, but won't make/receive any calls. Attached phones lose registration etc. ****** ADDITIONAL INFORMATION ****** Contents of a sample event_log.XXXXX file: Dec 10 18:28:40 asterisk[1219]: Restarted Asterisk Event Logger Dec 10 18:28:40 asterisk[1219]: Rotated Logs Per SIGXFSZ Contents of a sample messages.XXXXXX file: Dec 10 18:28:42 WARNING[1219]: Translated frame write failed (there are lots and lots of these files, all with only one or two lines in them)
Comments:	By: Nick Barnes (bcnit) 2005-12-13 06:46:47.000-0600 Many apologies - I missed a '1' off. This bug exists on version 1.0.9, not 0.9 as above. Again, sorry. By: Kevin P. Fleming (kpfleming) 2005-12-13 09:56:34.000-0600 This will fail to work properly in Asterisk 1.2.x as well. However, I cannot find any way we can easily deal with this, nor would it be an issue for any normal installation. Even recording raw audio (SLINEAR), it would take a 37 hour long call to hit the 2GB ceiling; using a compressed codec would make it take far longer. Given that, I can't agree that this qualifies as 'major' or is even technically a bug at all... I don't think it's a reasonable expectation that Asterisk would be able to monitor a 37+ hour long call. By: Denis Smirnov (mithraen) 2005-12-13 11:25:55.000-0600 Can Monitor() be used for writing video? By: Joshua C. Colp (jcolp) 2005-12-13 11:30:15.000-0600 No. By: Kevin P. Fleming (kpfleming) 2005-12-13 12:01:27.000-0600 On a somewhat more helpful note <G>, you can work around this problem by running Asterisk on a 64-bit platform using a 64-bit userspace... then the file size limit will go away (effectively). By: Nick Barnes (bcnit) 2005-12-13 14:05:01.000-0600 Firstly, let me say that I hear what you're saying and can understand where you are coming from. However, I am a little disappointed to read that our wanting to record a call lasting longer than 2 days is deemed unreasonable. To ensure that none of our sites experience this block in future, we will add a 'S(130000)' option in every Dial() command. I will update the Wiki accordingly. However, I really would appreciate a fix to this problem though - does Monitor() really have to block everything else if it fails? By: Tilghman Lesher (tilghman) 2005-12-13 14:10:20.000-0600 How about if you use the gsm file format to record your 42-day calls? By: Kevin P. Fleming (kpfleming) 2005-12-13 14:11:11.000-0600 Monitor() is not failing. What is happening is that the files are hitting the operating system's file size limit, and the kernel is sending SIGFSZ to Asterisk. When Asterisk receives that, it assumes it must be because a log file has hit that limit, so it rotates the logs. Asterisk is not prepared for _any_ other write operation to generate SIGFSZ. Because the continual writing is happening from Monitor() the log rotation keeps happening repeatedly, and Asterisk falls apart (essentially). The only reasonable fix I can see for this is to add a size limit to Monitor() so that if the file reaches that size it closes the file and starts a new one; if we do add that, it would also have to be added to MixMonitor() (which would be far more efficient for this application anyway, since it does the audio mixing on the fly). By: Tilghman Lesher (tilghman) 2005-12-17 13:51:50.000-0600 We could use off64_t and the related lseek64() in Linux, but that's not portable.  FreeBSD's off_t is already 64-bit.  Presumably, that size extends to the other BSDs as well. By: Tilghman Lesher (tilghman) 2005-12-17 15:16:00.000-0600 Another thing to note is that au, wav, and wav49 are all formats that are effectively limited to 32-bit lengths, due to a 32-bit length encoding in their headers. Headerless formats, such as g726, gsm, ilbc, alaw, ulaw, slin, and adpcm are all capable of extending beyond the 32-bit limit.  This is partially why I recommended that you use gsm for your recordings; the other reason is that it is a compressed format that should extend your call limit to 177 hours (a little over 7 days). By: Tilghman Lesher (tilghman) 2005-12-19 11:26:42.000-0600 This is an EXPERIMENTAL patch to enable 64-bit file offsets on a 32-bit architecture (Linux-only).  Please try it out and see if it does what you want.  The changes I have made are not oddball -- I have simply changed the types that various functions use from a 32-bit type to a 64-bit type and changed the appropriate library calls to use the 64-bit equivalents. The question won't be whether I have done something too far out that will make your system unstable; the question will be whether I have gone far enough.  The alternative is simply to have the same problem as described in this bug, which is why I encourage you to try this out. By: Tilghman Lesher (tilghman) 2005-12-19 11:27:43.000-0600 Oh, and note that due to the reason outlined above, if you attempt to use au, wav, or WAV as your filetype, weird things are likely to happen. By: Tilghman Lesher (tilghman) 2005-12-29 15:54:12.000-0600 I need your response to tell me if this patch fixes your issue or not. By: Jason Parker (jparker) 2005-12-30 14:36:07.000-0600 Tested patch using VoicemailMain (option 3, then option 5 to leave a message) with odbcstorage.  (it probably used wav49) It was able to create and save the 7 second file, but when it went to play it back, it acted as though the file was 0 seconds long. I then reverted the patch and tested again and it worked as expected, so it was indeed caused by this patch. By: Matt O'Gorman (mogorman) 2006-01-11 18:13:00.000-0600 okay, not to be a nay sayer, but I have to agree with kevin in that this is not needed.  If you need to have longer than 2 day calls a compressed codec would do.  I agree that a monitor rotate would be a valid option as well.  anyways wht do yall think? By: Tilghman Lesher (tilghman) 2006-01-11 18:28:01.000-0600 Well, the thing is that we have support for 64-bit file offsets, even in a 32-bit kernel, so we really should be using them.  The fact that a 37-hour recording is ridiculous is beside the point. Actually, I suppose we should enable 64-bit file offsets just for the logfiles, as we would benefit there, but you can't enable 64-bit file offsets for just some files and not others.  It's an all-or-nothing deal. By: Matt O'Gorman (mogorman) 2006-01-11 23:23:45.000-0600 it was my understanding that these changes would not work with our other posix compliant brothers.  I think the better option is a log rotate for files. By: Tilghman Lesher (tilghman) 2006-01-11 23:29:40.000-0600 These changes are compliant with SUSv2, which is the successor to POSIX.  FreeBSD 3.5 (2000) already had both of the required functions fseeko() and ftello(), so compliance should not be a problem. By: Nick Barnes (bcnit) 2006-01-12 04:58:04.000-0600 I agree that a compressed codec *should* be fine for long calls, but if you don't know in advance how long the call is going to be and you really want 99.99% of your calls to be recorded uncompressed, how can this be achieved? If I had the option of changing codec if the call looked like it was going to be too long, then fine, but I suspect that this would be more complicated than rotating. But I'm still not convinced that it is acceptable behaviour for Asterisk to bomb out completely if only one call goes on too long ;-) By: Kevin P. Fleming (kpfleming) 2006-02-13 16:57:16.000-0600 OK, let's put this one to bed: I'm ok with changing to 64-bit off_t across the board, but we need a tested and working patch. By: Denis Smirnov (mithraen) 2006-02-19 10:45:25.000-0600 before using _FILE_OFFSET 64 we can apply patch from 0006540 that only change types and use fseeko/ftello. By: Tilghman Lesher (tilghman) 2006-02-20 13:30:16.000-0600 Patch updated to current trunk, changing the single line to activate 64-bit offsets to off.  Again, au, wav, and wav49 will never support larger sizes than 4GB, due to header constraints. By: Tilghman Lesher (tilghman) 2006-02-20 16:36:41.000-0600 Patch tested and committed to trunk.