[Home]

Summary:ASTERISK-01213: Very Bad Audio (Jitter) Sip-to-Sip
Reporter:bfranks (bfranks)Labels:
Date Opened:2004-03-14 22:28:51.000-0600Date Closed:2004-09-25 02:52:17
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) codec_g723_1.c.diff
( 1) translate.c.diff
Description:The audio between two SIP hardphones (Polycom IP500's) is very bad when using latest (March 13) CVS.  I saw in the IRC channel the same problem occurs with Cisco 7960's.

After downgrading back to January 12 CVS, the problem was resolved.

****** ADDITIONAL INFORMATION ******

I am using canreinvite=no and the problem makes a phone call unbearable.

Using the Ulaw codec, also tried Alaw and same problem.

SIP-ZAP (using Ulaw, T100P and Adtran CB) the calls are fine, however when SIP-to-SIP (Internal) calls are placed the audio quality is horrible.

Ruled out everything (network Quality, specific phones) and problem appears to be with the latest CVS.
Comments:By: bdolljr (bdolljr) 2004-03-15 12:30:15.000-0600

Same problem here...  Except my SIP-ZAP (using U-LAW, TE410P) is horrible as well.  Unusable.  I have 7960's.
By: heison (heison) 2004-03-15 19:58:04.000-0600

With the latest CVS (2004-03-15 1900EST), I had the same problem on my 7960's, but the 2 budgetone BT102's seem to survive. The analog phones on TDM400 was not affected.

Downgrading to 2004-03-12 works well.
By: Brian West (bkw918) 2004-03-16 11:59:10.000-0600

I have two 7960's and two X100P's I have NEVER seen these problems at any point.
By: Mark Spencer (markster) 2004-03-16 12:15:30.000-0600

This should already be fixed as of Sunday.
By: Mark Spencer (markster) 2004-03-16 12:16:39.000-0600

Also, can you confirm whether this problem is in stable, head, or both?
By: bfranks (bfranks) 2004-03-16 12:17:32.000-0600

Will upgrade this weekend when I can schedule downtime and will report on the results.
By: bfranks (bfranks) 2004-03-16 12:18:26.000-0600

I did an upgrade from head.  This weekend I will try both head and stable.
By: fbnts (fbnts) 2004-03-16 18:23:57.000-0600

I experienced this problem with a recent install of the development version of Asterisk.  I have cured it by downloading the stable Branch
By: heison (heison) 2004-03-16 18:28:28.000-0600

I did 'cvs update -A' with my 2004-03-12 tree, everything seem to be working still. I'm at current 2004-03-16.
By: senad (senad) 2004-03-17 09:24:18.000-0600

I have similar problems as well. It started after rtp.c file was changed to fix timestamp issue. ;).

With my setup SIP>SIP work fine.
SIPURA > SIP > * >SIP > SIPURA

IAX:
Sipura > SIP> * > IAX2 > * ... is HUGE quality problem

So.. is :
SIPURA > SIP> * > H323/G729 > Cisco 5300


Message "Difference is XXXX" has now increased in number of digits to XXXXXXXX

:)...
By: dwildes (dwildes) 2004-03-17 10:02:37.000-0600

I agree - the problem happens between revision 1.52 & 1.53 or rtp.c

1.52 works fine - 1.53 creates VERY choppy audio
By: dwildes (dwildes) 2004-03-17 10:52:50.000-0600

I found the line in rtp.c that is causing the problem:
from CVS HEAD tag 1.57, line 997:

if you change it from:
if (!f->delivery.tv_sec && f->delivery.tv_usec) {

to:
if (!f->delivery.tv_usec) {

You don't get the jitter/choppy audio anymore.
So the problem appears to lie within the checking of f->delivery.tv_sec
By: senad (senad) 2004-03-17 11:39:47.000-0600

i did above.. and it fixed the problem..
Thanks... :)
By: Mark Spencer (markster) 2004-03-17 11:54:58.000-0600

Actually I believe the correct fix is:

if (!f->delivery.tv_sec && !f->delivery.tv_usec) {
}

Feel free to confirm that fixes it, i've dumped it in cvs head.  Good eyes!
By: dwildes (dwildes) 2004-03-17 12:02:07.000-0600

Thanks - I've tested the updated syntax & it works perfect for me!
By: zoa (zoa) 2004-03-17 12:06:18.000-0600

fixed in cvs
By: heison (heison) 2004-03-22 10:36:05.000-0600

The problem is reproducible with my 7960.

7960 <-> Zap : OKAY
7960 <-> IAXy : bad audio
7960 <-> Asterisk AGI (weather.agi) : bad audio

I have double checked my CVS, it does have the fix in rtp.c:

999: if (!f->delivery.tv_sec && !f->delivery.tv_usec) {
By: Mark Spencer (markster) 2004-03-22 11:21:33.000-0600

What codec are you running?
By: heison (heison) 2004-03-22 11:52:34.000-0600

Problem persist with both G729 and G711. Note, it works fine on GS phones.
By: Paul Cadach (pcadach) 2004-03-22 12:10:24.000-0600

Please, don't use G.729 with new frame structure (which having delivery member field) because of size differences in pre-compiled at codec_g729b.so and in Asterisk until any sort of workaround will be available. Currently you could destroy codec's internal structures...
By: Mark Spencer (markster) 2004-03-22 12:14:30.000-0600

With latest CVS i cannot duplicate this problem running ulaw from iaxy to snom phone, and the timestamps look just fine on ethereal, so i don't know where your problem is coming from.  you can try to find me on irc and discuss it if you like.
By: bdolljr (bdolljr) 2004-03-22 13:47:03.000-0600

I believe the problem may be specific to the 7960.  My 7960's are running SIP - 5.3 firmware and I am able to reproduce bad quality audio very easily with the following test.

7960 <-> Milliwatt()

My 7960 is using u-law codec only as I don't have any g.729 on my system.

With rtp 1.5.2 this works fine.  With rtp 1.5.3 it's very broken even with the fix:

999: if (!f->delivery.tv_sec && !f->delivery.tv_usec) {
By: Mark Spencer (markster) 2004-03-22 14:47:12.000-0600

Millwatt was updated very recently, and there was a problem specific to milliwatt.  Please be sure you have the very latest CVS on your system.
By: bdolljr (bdolljr) 2004-03-22 17:44:14.000-0600

OK. I upgraded to -devel CVS 03/22/04 and Milliwatt() is indeed fixed. :}  Unfortunately, I am still having quality problems that were not there with the previous rtp.c versions.  Specifically, one example is:

7960 u-law -> * -> IAX2 gsm -> * -> IAX2 gsm (Digium 17004286000)

I can make my box(es) available via SSH if anyone wants to take a look.  AIM:bjrbigsky
By: heison (heison) 2004-03-23 00:05:48.000-0600

I've tried rtp.c 1.52, 1.60 & 1.61, I experience the same quality problem. When I down graded to 1.51, it seems to work fine.
By: heison (heison) 2004-03-24 07:59:46.000-0600

With head CVS, 7960 u-law -> * -> IAX2 g729 -> nufone -> PSTN also gives poor quality.
By: Paul Cadach (pcadach) 2004-03-24 10:58:15.000-0600

I have problems with 7960 too, but situation is very interesting.
Path 7960 g.729 -> * under VMware -> IAX2 g.729 over V.34 modem -> * -> g.723.1 over the same V.34 modem -> Cisco-12SP+ works well but reverse direction gets very ugly sounds. Also, path 7960 g.729 -> * under VMware with Echo app -> same 7960 g.729 works well, but 7960 g.729 -> * under VMware -> IAX2 g.729 over V.34 modem -> * with Echo app and back works bad.

Those schemes points that 7960 looks like very sensitive for timestamping, while Cisco-12SP+ works fine with really ANY timestamps (it works well with sequence of packets stamped with 160 then 320 RTP's ticks, but g.723.1 data is equal to 240 ticks at each packet).

I'm trying to figure out 7960 g.729 problems at shown connection scheme. If any news (good or bad) will be available I'll notify here.
By: zoa (zoa) 2004-03-24 15:01:38.000-0600

my 7960 ulaw -> iax2 gsm -> pstn also sounds baaaad with latest cvs.
pstn -> iax2 gsm -> pstn sounds normal.
By: zoa (zoa) 2004-03-25 03:01:03.000-0600

i also have delayed sound with xlite.
By: zoa (zoa) 2004-03-25 04:05:05.000-0600

i upgraded both asterisk servers to cvs of today.
When i call from xlite to * -> iax2 -> *(echo test), the sound is e x t r e m e l y    d e l a y e d.
By: Mark Spencer (markster) 2004-03-25 17:23:56.000-0600

echo test is a *bad* test.  Can you please try calling something else?  I'll work on app_echo in the mean time.
By: zoa (zoa) 2004-03-26 10:24:43.000-0600

normal calls with xlite sound normal.
(they don't have the problem the cisco has)
By: Ricardo Villa (ricvil) 2004-03-27 19:37:32.000-0600

To answer heison's comment that audio is fine on the GS phones:

It is important to note that the TimeStamp issue does not affect less sophisticated phones like the Grandstream.  It only affects the ones that have elaborate jitter buffer algorithms that rely on the TimeStamp value, like the SPA2000 and probably the Cisco's too.
By: rburkholder (rburkholder) 2004-03-27 21:18:25.000-0600

Perhaps the fix mentioned in 1312 may solve the problem.
By: Mark Spencer (markster) 2004-04-02 02:31:20.000-0600

Okay, is this still a problem?  I cannot duplicate this, nor can most people that i know.  one person had the problem because one of the two servers hadn't been updated yet.
By: zoa (zoa) 2004-04-02 03:00:35.000-0600

i had it with both servers updated. (i'll try again with latest cvs when my server is back in the colo).

The problem with the echo program being slow is resolved.
By: Ricardo Villa (ricvil) 2004-04-02 09:36:26.000-0600

The SIP-SIP issue seems to be resolved for us.  But the SIP-IAX2 is still problematic.  All our servers were updated to CVS from 3/30/2004.  Communication between our servers whether its SIP-SIP or SIP-IAX2-ZAP is perfect.  But once we try do SIP-IAX2 with Nufone, the audio is horrible on the PSTN side but perfect on the SIP side.

Greg and Jeremy tell me they have no problems on their side.  So I can't figure out where the problem can be.  Any ideas?
By: Mark Spencer (markster) 2004-04-02 11:24:37.000-0600

Can you look at the timestamps under ethereal and see if they're being messed up?
By: Ricardo Villa (ricvil) 2004-04-02 16:06:50.000-0600

On our end (the SIP end) the TimeStamps are normal (except for the ocasional jump of 152 or 168 samples, which does not affect the call that much).  The audio on our side is of very good quality.  

To view Timestamps on the IAX2 side (the one we are transmitting to Nufone), is there an Ethereal version that can display them?  I remember somebody mentioning some decoder plugin being developed but I don't know if its in any version yet.  

Otherwise how would I see the TimeStamps on this segment of the call?
By: Mark Spencer (markster) 2004-04-02 16:09:57.000-0600

yes, there is one somewhere.  google is your friend.
By: Ricardo Villa (ricvil) 2004-04-02 17:59:56.000-0600

OK..I found the IAX2 decoder with the Ethereal CVS Version.  A test call from one of our servers out to Nufone shows the incoming TimeStamps at 20ms and 40ms.  They seem to alternate between 20ms and 40ms one packet after another.  The audio on our end is great (SIP Phone).  The TimeStamps on the outgoing packets show the same behavior, 20ms, then 40ms, then 20ms, etc... The Audio on the PSTN side (Florida) is horrible.  Every word seems to have a click or one letter missing.

Codec used in this test is iLBC.  Is that the proper TimeStamp behavior?  If I use Asterisk Version 0.7.2 instead, then the outgoing TimeStamps are all over the place.  Not neatly at 20ms and 40ms.

Let me know what else I can check.
By: Mark Spencer (markster) 2004-04-02 18:28:28.000-0600

This is totally about iLBC.  Can you confirm that going to GSM makes the problem go away?  If so I'd like to close this bug and open a new, specific one.
By: Ricardo Villa (ricvil) 2004-04-02 19:14:05.000-0600

I have tried GSM and uLaw and both produce the same kind of bad audio on the PSTN side.  I do not think this problem is codec dependent.  In the case of uLaw I do see that both the incoming and outgoing TimeStamps are evenly spaced at 20ms.
By: Paul Cadach (pcadach) 2004-04-02 22:44:52.000-0600

Try attached patch (translate.c.diff)... This one allows to adapt timestamps when you have translation between codecs with diffirent frame sizes (for example, G.711/G.729 frame is 20 ms while G.723.1 frame is 30 ms). Requires to have VALID f->samples values (see codec_g723_1.c.diff).
By: Mark Spencer (markster) 2004-04-03 21:05:10.000-0600

So does your patch fix it?
By: Ricardo Villa (ricvil) 2004-04-03 23:15:12.000-0600

To keep the test as simple as possible I did not use the patch.  I retested by using uLaw on our phone and on the IAX2 connection to Nufone, so frame sizes were all the same.  The results were the identical.  Perfect Audio on our end and horrible audio on the PSTN side.

Then I compared by making calls via IconnectHere.  The audio was crystal clear in both directions.  The problem was then isolated to the IAX2 connection with Nufone.

Our Server is on a dedicated T1 and all tests were done with no traffic on the LAN.  Only the test call was using bandwidth.

I then started to take a close look at jitter statistics on Ethereal.  The office next door where the phone is has an ADSL connection.  The incoming RTP stream (from our SIP phone as seen on the server) showed a an average jitter of about 10ms, and thus the jitter going out on the IAX2 link was about the same.  The audio was subpar.  If we caused the jitter to go up to about 25ms then the audio became severely degraded on the PSTN side, but still perfect on the SIP side.

So now that we had jitter as a suspect, we decided to plug in the phone directly on the T1 connection with the server.  The jitter dropped to less than 1ms.  The audio was now perfect in both sides.  

The conclusion here is that in order to work with Nufone, you need an outgoing jitter that is ureasonably low.  I think that it is extremely hard to obtain a jitter close to 1ms unless you have a T1.  

Maybe Nufone has the jitter buffer disabled on the connection we are using.  I will report this to them now.  Nufone used to work fine for us before.  Now we are only using Iconnect until we sort this out.
By: Paul Cadach (pcadach) 2004-04-03 23:21:56.000-0600

It must fixes timestamping when converting between codecs with different frame sizes (iLBC have 30 ms frame while GSM have 20 ms frame). Exact sequence is (all timestamps are relative):
1) iLBC frame timestamped at 0 ms produces GSM frame 20 ms with the same timestamp, but 10 ms of audio stay on hold at codec level;
2) iLBC frame timestamped at 30 ms produces double GSM frame 40 ms with the same timestamp (30 ms) while one must be timestamped with 20 ms.

Also, g.723.1 codec erroneously assumes that frame->samples field is in ms rather than in samples, so all values must be multipled by samples per ms values (i.e. 8 samples per ms).
By: Mark Spencer (markster) 2004-04-04 11:07:47

I think there is more to the story here.  rcivil: When you run with 25ms of jitter, do you see that jitter reflected in the IAX2 numbers?  i.e. the IAX2 number should still have proper numbers in them, and should NOT duplicate the jitter that is on the network from you to the SIP phone.  Can you confirmt hat is really the case?
By: Ricardo Villa (ricvil) 2004-04-04 19:35:32

I don't see 25ms of jitter on the incoming IAX2 stream.  The received IAX2 stream from Nufone shows jitter at around 4-6ms.  The audio is pretty much perfect and we also have our jitter buffer enabled.  

But by looking at the outgoing stream of IAX2 Ethereal traces on our server we can guesstimate that as the jitter grows in the incoming SIP RTP stream, so does the jitter on the outoing IAX2 stream.  I look at the datestamps of outoing packets to reach this conclusion (it does look to be somewhere in between 20-30ms, which is similar to what the incoming SIP RTP stream shows.  I thought this seemed reasonable to me since I do not think that the Astersik SIP channel has a jitter buffer.  So any jitter received simply gets carried over to the outgoing channel.  Am I mistaken in this assumption?  If so please let me know how to enable a jitter buffer on the incoming SIP RTP Stream since this would most likely fix our issue.
By: Mark Spencer (markster) 2004-04-04 23:54:46

I think we're not talking the same thing yet.  My question is whether the timestamp field of the packets going across the IAX2 link from you to nufone internally (irrespective of the actual time they are transmitted) show themselves to be increasing by 20ms or not.
By: Ricardo Villa (ricvil) 2004-04-05 10:47:27

Yes they do.  Timestamps are mostly increasing by 20ms or so.  Every now and then I see a jump of 21ms or 19ms, but I already pointed this out before.  On the other hand the actual datestamp I see on Ethereal is not increasing by 20ms.  I see a lot of jitter.  Some packets are spaced 1ms apart, some 40ms, some 30ms, etc...  It is comparable to the jitter on the incoming SIP RTP Stream.  Roughly between 20ms and 30ms on average.  This leads me to believe that Nufone's iax.conf has the jitter buffer disabled (I am asking them about this).  Otherwise I should be getting good audio on the PSTN side.  At least it works fine between all our servers with IAX2 links between them.

edited on: 04-05-04 09:39
By: Mark Spencer (markster) 2004-04-05 12:57:00

Okay so I think if it's working with your SIP -> IAX2 -> SIP environment it's probably safe to close this one out as some sort of configuration issue or termination issue at nufone's end, but feel free to open a *new ticket* if this ends up being an Asterisk issue again.  This ticket has diverged very far from the original concept.
By: Mark Spencer (markster) 2004-04-05 15:13:38

Fixed in CVS