Summary:	ASTERISK-05672: [patch] Monitor() recordings have 12dB too much gain
Reporter:	John Walliker (jrwalliker)	Labels:
Date Opened:	2005-11-22 06:53:50.000-0600	Date Closed:	2007-04-05 18:15:35
Priority:	Minor	Regression?	No
Status:	Closed/Complete	Components:	Core/CodecInterface
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) calibration-tones-18-12-6-0-8kHz.wav ( 1) format_wav.c.diff ( 2) format_wav2.c.diff ( 3) recording-test-tones-out.wav
Description:	Recordings made using Monitor() have 12dB (4x) gain compared with the incoming signal.  This happens with 1.2.0 and with 1.0.x running on Fedora core 3 or 4. The 12dB gain means that speech is often clipped.  If the output of Milliwatt() is recorded, it is severely clipped.  For example: exten => 174,1,Answer() exten => 174,2,Monitor(wav,/jrwtest/recording) exten => 174,3,Milliwatt() exten => 174,4,Stopmonitor() exten => 174,5,Hangup() When wav files are played with Background() the same thing happens.  For example: exten => 174,1,Answer() exten => 174,2,Monitor(wav,/jrwtest/recording) exten => 174,3,Background(/jrwtest/calibration-tones-18-12-6-0-8kHz) exten => 174,4,Stopmonitor() exten => 174,5,Hangup() I have attached the 16-bit linear test file I used which has tones at 0dB, -6dB, -12dB and -18dB relative to clipping level. Other users have been able to replicate this problem.  I think it may also explain some complaints on asterisk-users of "choppy audio" on Monitor() recordings. Other users have duplicated the problem. John Walliker
Comments:	By: Kevin P. Fleming (kpfleming) 2005-12-12 20:27:50.000-0600 Please post some details of how you made these recordings (channel type and settings). The monitoring process does _not_ have the ability to adjust the signal level during recording, so I don't understand why this would be happening. By: John Walliker (jrwalliker) 2005-12-13 07:19:42.000-0600 I am running two systems at the moment.  (I will set up a third system, also using a Pentium-M with 1.2.1 on Core 4 shortly.) System 1 uses a DFI G5M100-N motherboard with a Pentium-M processor and is running 1.2.0 on Fedora Core 4.  It receives calls using IAX2 over the internet from two different servers through an ADSL broadband connection. System 2 uses a Via EPIA M10000 motherboard with a Billion BIPAC-PCI (hfc chipset) card and is running BRIstuff 0.3.0-PRE-1b on Fedora Core 3.  It receives calls using an ISDN2e line. Both systems show the problem.  To demonstrate the problem, I make a call into the system which causes either Milliwatt() or Background() to play a test signal.  The first file I posted is the test recording I used in conjunction with Background(). The recording made by Monitor() shows the excess gain.  The second file I posted is an example of such a recording with too much gain.  The signal received by the caller over IAX2 or ISDN2 is fine in both cases. System 1 is probably easier to analyse, so here are the relevant .conf files. iax.conf contains the following: [general] iaxcompat=yes disallow=all allow=alaw jitterbuffer=yes forcejitterbuffer=no tos=lowdelay autokill=yes [guest] type=user context=default [jwcl] type=user host=xxxx secret=???? context=default extensions.conf contains the following: [general] static=yes writeprotect=no autofallthrough=yes clearglobalvars=no priorityjumping=no [globals] CONSOLE=Console/dsp IAXINFO=guest TRUNK=Zap/g2 TRUNKMSD=1 ... examples ... [default] exten => 174953,1,Answer() exten => 174953,2,Monitor(wav,/jrwtest/recording) exten => 174953,3,Background(/jrwtest/calibration-tones-18-12-6-0-8kHz) exten => 174953,4,Stopmonitor() exten => 174953,5,Hangup() exten => 174954,1,Answer() exten => 174954,2,Monitor(wav,/jrwtest/recording) exten => 174954,3,Milliwatt() exten => 174954,4,Stopmonitor() exten => 174954,5,Hangup() By: Kevin P. Fleming (kpfleming) 2005-12-13 09:39:22.000-0600 format_wav.c does explicit gain adjustment when reading/writing WAV files, but I don't know why (although I'll try to find out). It assumes that it will only be used to read/write files that it created itself, so when you supply a file from an external source that adjustments are not 'transparent'. I suspect if you listen to the recorded file using Asterisk itself, instead of an external tool, you'll find that the gain adjustment disappears... By: Kevin P. Fleming (kpfleming) 2005-12-13 16:56:44.000-0600 OK, I have confirmed with Mark that the gain adjustments in format_wav.c are intentional, and are designed to compensate for the fact that the telephone network and PC audio systems don't have the same zero-gain playback level. None of the other file format drivers (including WAV, which uses GSM-inside-WAV) do this gain adjustment, so the audio out would be the same as the audio in. Please note that you are only noticing this because the wav input file was _not_ created by Asterisk, so it doesn't downward adjust the signal at read time (as it would if it thought it had created the file). Normally the read/write operation would result in no level change, but since the file was not created by Asterisk that doesn't occur here. Is there an actual problem you are working on besides this specific test case? By: John Walliker (jrwalliker) 2005-12-13 17:17:11.000-0600 The problem also happens when I record the output of Milliwatt().  If you run the example I gave above, the recorded output of the Milliwatt() tone is severely clipped and looks more like a square wave than a sine wave. There is a real problem.  I am developing a test system based on Asterisk for measuring the audio quality of various terminal devices.  I really do want to record exactly what is going in or out over the audio link.  I am used to being able to make bit-exact audio transmissions across ISDN and into or out of PCs with CAPI interfaces on ISDN2.  I was hoping to extend this to multiple E1 connections by using Asterisk.  I will try rebuilding Asterisk with gain=0 to see if this does exactly what I want.  If so, it would be nice to have a .conf option somewhere to set this as I know there are other people who would benefit. Many thanks for looking into this. John By: John Walliker (jrwalliker) 2005-12-22 08:09:09.000-0600 I have been testing for several days now with gain set to zero in format_wav.c.  I also tried removing all the gain changing code for reading and writing wav files to avoid converting to float and back again and unnecessarily copying data. In both cases, the monitor recordings of incoming audio, outgoing Milliwatt() and outgoing Background() files appear to be correct. I have come across some odd but very subtle effects during testing, but these need further investigation and should perhaps be reported separately. (It appears that some unnecessary transcoding from slin->Alaw->slin or slin->GSM->slin (depending on which codec is selected for incoming IAX) is going on between playback of Background() or Milliwatt() and the recording made using Monitor.  I see this (in the form of transcoding artifacts) in a system using 1.2.1 and IAX but not on another one running 1.2.0 BRIstuff and isdn2.) I have also been thinking more about the gain issue.  It seems to me that the default should really be zero, because this preserves all information in the recording.  In contrast, the current setting of gain=2 slices off the top 12dB of the dynamic range and any high level information is lost forever.  The dynamic gain adjustment code cannot restore what has been chopped off.  I don't have any problem getting the recordings to play loudly enough with gain=0. I suppose there is a backwards compatibility issue for those who have a lot of existing recordings... Any comments? John By: Matt O'Gorman (mogorman) 2006-01-13 13:52:13.000-0600 Almost a week or so ago, we fixed a few issues with the monotoring code where there are clicks and pops in the audio stream.  I reccomend that you guys retry this.  I also think that it is a design decision for wav to be amped for pc use, where as WAV does not have an issue, I do think that someone submitting an adittion to the doc folder, a README.monitor that explains this. thanks Mog By: John Walliker (jrwalliker) 2006-01-13 17:04:06.000-0600 Thanks.  I will run some more tests with the latest code next week.  I think a README is a good idea - I would be happy to write one if it has not already been done. I still think that it would be better to record the audio accurately and then increase the gain afterwards if necessary.  I feel very uncomfortable about doing something as a default which permanently removes the signal peaks.  Maybe audio levels in the UK tend to be higher than elsewhere - I have certainly found that a large proportion of phone calls have their peaks clipped by monitor().  Most of the recordings I make originate from GSM mobile phones - perhaps this is another reason why I see a lot of high level signals. Can it really be right that Asterisk generates a test signal with milliwatt() which cannot be recorded by monitor without clipping? John By: kb1_kanobe2 (kb1_kanobe2) 2006-02-01 20:43:37.000-0600 To add another perspective to this, I'm quite worried that the supposedly raw wav format is messing with gain. I have relied on monitor() based recordings using the uncompressed wav format in the past as a source for testing echo canceller code changes. wav was chosen specifically because it most accurately represented the signal on the wire and hence could be spooled through the echo can block by block during performance testing. I strongly suggest that if the intention is to preserve this additive gain behaviour that it be clearly exposed to the dialplan, perhaps through a defaulted optional parameter, thereby allowing complete transparency and control over. Just my 2c. k. By: Andrew Pollack (andrewjaypollack) 2006-02-16 05:38:54.000-0600 Just a note from someone also seeing choppiness (if that's a word) when using monitor.  I attributed it to the gain issue and tried setting gain to 0 as described here (which I THINK means that the gain adjustment increments in steps of 0 now -- which in practices means it doesn't -- rather than starting at 2 and going up in 2's). I did not find that it resolved the issue in any way for me personally.  I'm only adding this note so that others who come across this thread do not assume this is "the" resolution.  Perhaps its just "A" resolution for one person. Forgive me, here, if I'm breaking protocol.  My "C" Kung-Fu is rusty, and I'm new to this Dojo. By: Serge Vecher (serge-v) 2006-05-06 14:17:51 any takers for document this issue in a README file? By: John Walliker (jrwalliker) 2006-05-06 18:37:45 OK - I'll write a readme within the next few days. John By: Serge Vecher (serge-v) 2006-05-23 10:28:09 john: a related issue has been closed due to "the issue is not there anymore" in 1.2. If there is anything that needs to be done to the code, in terms of a README or a patch, please feel free to reopen the issue with the goodies to look at. Thank you. By: timrobinson (timrobinson) 2006-11-10 07:44:39.000-0600 We have recently had the same issue here.  I am somewhat at a loss as to why we are tweaking the recording gains at all, as the top 12 dB are basically clipped! This makes no sense to me, and the explanation of 'compatibility with Windows' does not justify adding so much gain that many recordings are badly distorted. I have made a change to set #define GAIN 0 and have briefly tested it - recordings are no longer distorted. I would strongly recommend that the default should be zero gain on all recordings so as not to deliberately add distortion to a digital signal. By: Nic Bellamy (nic_bellamy) 2006-11-11 16:16:59.000-0600 I second TimRobinson - and indeed I change the gain to 0 on all systems I deploy. From what I can tell, the original reasoning behind it is for voicemail - when sending a PCM .wav file out, it sounds very quiet on a PC, as PSTN voice levels tend to be more around the -12 to -25 dBFS level. However, corrupting (that's what I'd call it) the stored audio in the core is a bad bad idea. If people want louder voicemail messages when they're listening on their PC, the way to do this would be to keep the stored audio accurate, and normalise the version that is emailed. I guess I should write a patch, eh? :-) By: John Walliker (jrwalliker) 2006-11-12 17:03:52.000-0600 I have been using #define GAIN 0 since last December with many different builds of * and it always works fine. I continue to believe that this should be the default, but it has been difficult to convince others that this change should be made.  Perhaps this is a good moment to try again! John Walliker By: Serge Vecher (serge-v) 2006-11-17 09:05:48.000-0600 alright, so the consensus here is: let's write the patch and get people to test it? By: Serge Vecher (serge-v) 2006-11-17 09:07:02.000-0600 TimRobinson: do you have a patch to test, please? By: Ronald Chan (loloski) 2006-11-17 11:30:55.000-0600 serge: i belive this is the source they are pointing out ?? please correct me if i'm wrong :) Ronald By: John Walliker (jrwalliker) 2006-11-17 12:03:17.000-0600 format_wav.c.diff looks fine.  Perhaps it would be worth adding a comment to say that the default was previously 2, giving a gain of 4x or 12dB but clipping the peaks of loud signals. John By: Ronald Chan (loloski) 2006-11-17 12:20:23.000-0600 Ok, upload new patch with new comments :) By: John Walliker (jrwalliker) 2006-11-17 12:57:45.000-0600 I think the original description of what GAIN does should be retained! Also, the default value WAS 2, not IS 2 (once the patch has been applied). John By: Ronald Chan (loloski) 2006-11-17 13:36:09.000-0600 John: sorry :) i'm not a native english speaker, could you please send along your patch with proper comments since i can't construct it the way it should :) and hope for some developer to pick this up many thanks!!! Ronald By: John Walliker (jrwalliker) 2006-11-17 14:24:35.000-0600 OK - I've got no excuse now.  I'll write a detailed comment which explains what format_wav.c does when writing AND reading.  It does have some interesting properties... Its time to go now - kick me if I haven't done it by Sunday evening! John By: Serge Vecher (serge-v) 2006-11-20 15:27:56.000-0600 guys, please don't forget to mention your disclaimer status when uploading patches into the bug issue that is not marked as 'yours'. Thanks. By: Anthony LaMantia (alamantia) 2007-01-04 16:49:01.000-0600 *Kicks* jrwalliker ;) By: Dwayne Hubbard (dhubbard) 2007-04-05 18:13:22 Patch is merged.  Thanks everyone for their comments.  For those who might have needed the original value of GAIN for voicemail, please refer to the 'volgain' option in voicemail.conf.