Summary:	ASTERISK-02952: IAX2 to IAX2 huge delays
Reporter:	zoa (zoa)	Labels:
Date Opened:	2004-12-09 14:07:45.000-0600	Date Closed:	2011-06-07 14:04:45
Priority:	Major	Regression?	No
Status:	Closed/Complete	Components:	Core/General
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:
Description:	After several complaints i heard, i tried and indeed there is a problem with iax2 to iax2. Its easy to test, just dial with iaxcomm to * to iaxcomm. Very huge delays are heard.
Comments:	By: zoa (zoa) 2004-12-09 14:08:49.000-0600 just confirmed this bug, im on cvs-stable, others are on cvs-head and have the same thing. jitter buffer is disabled. By: tan (tan) 2004-12-09 14:10:41.000-0600 We can confirm this occurs on the latest cvs as well, and with Diax client as well. By: zoa (zoa) 2004-12-09 14:11:43.000-0600 its not in 100% of all cases, made 3 calls, one out of three had a huge delay. By: zoa (zoa) 2004-12-09 14:14:21.000-0600 will try to check this some more tomorrow, Btw, i also checked this on the local lan, on the first call i made there was a 30 seconds delay. No idea where this comes from. By: Mark Spencer (markster) 2004-12-09 14:43:58.000-0600 Does this happen if you go * -> * -> * instead of iaxcomm -> * -> iaxcomm In other words, how do you know this is a * bug and not an iaxcomm one? By: tan (tan) 2004-12-09 15:20:53.000-0600 We have * --> * --> * and haven't seen this issue. We have tried this with a number of iax clients and have seen the same issue. However, if they all use the same core then it might be that the core has a problem. How do we progress? By: Mark Spencer (markster) 2004-12-09 15:41:41.000-0600 Unless you have other information to the contrary, I do not believe this to be an Asterisk bug but an iax-client bug.  Please take this to the iax client mailing list.  THanks! By: zoa (zoa) 2005-03-31 09:08:58.000-0600 Looks like it needs patching on one of both ends. iaxclient to cvs-head to cvs-head to pstn gives delays the audio coming from the PSTN mic. IAxclient to cvs-stable to cvs-head to pstn has no issues at all. We used a iaxclient version of some days old (not the latest, which has an issues with the first seconds of the call being gone). It has been an issue for quite some time now, a lot of the activity on the iaxclient list is about this problem. According to stevek, tzanger was going to produce a patch for this. (dont know if it would be an iaxclient or an asterisk patch. But an asterisk patch seems possible anyhow. I dont have the resources to look into this now. (still working on the jitterbuffer). By: stevekstevek (stevekstevek) 2005-03-31 10:14:48.000-0600 (goddamnit, I just spent 10 minutes typing into this damned form, and accidentally hit back, and lost it.  That bites.  Now I'm using vi, and I'll paste it in afterwards). [getting over being pissed off at damned computers...] OK, the main problem that I've seen that leads to this is that asterisk sometimes generated timestamp sequences with discontinuities (i.e. jumps).  The jitterbuffer sees this as a big jitter (and either very early or very late frames), and you end up with a big delay. I think we ought to solve this in two ways: (1) In the jitterbuffer, somehow detect these discontinuities, and adapt to them. (2) Get asterisk to not do this anymore.  It's sending us incorrect information, and it's an error. (1) is difficult, because there's no absolute way to detect the difference between these jumps, and real network delays.  You can come up with a heuristic which will work most of the time, and cause a resynchronization when the jumps are big, but you'll still have cases where the jumps are small and they're not detected, and some cases where there's real network issues and a resync happens (leading to gaps and such in the audio).  Also, choosing this threshold probably needs to be dynamic and/or configurable;  something like (2*(measured jitter) + 1000ms) or something, so that when there's real jitter, we have a higher threshold than where we haven't previously seen jitter. (2) is more straightforward, at least for the common case that I've seen so far.  The issue here is that asterisk channels (I'm mainly talking about chan_iax2, which I know the best, but I think that chan_sip and the others work basically the same way) put timestamps on outgoing frames in one of (three) ways: a) non-voice frames:  Timestamps are generated by the channel driver, using the present time of day, and the pvt->offset member. b) voice frames with "implicit" timestamps:  These are voice frames that get to the channel driver with no frame->delivery set;  These types of frames come from most apps, ast_stream, and bridged non-VoIP channels.  Here, we use "timestamp prediction", to generate nice accurate timestamps, clocked to the source of the frames, and also adjust pvt->offset such that the entire conversation is clocked by the source of these timestamps. c) voice frames with "explicit" timestamps:  These are voice frames that get to the channel from a bridged VoIP channel (where the timestamps that come in are based on that channel's "rxcore" and the frame's incoming timestamp), or from some apps which timestamp outgoing frames (like app_conference, which does this so that the receiver's jitterbuffer can compensate for scheduling delays in the conference processing). This also happens (using a different codepath) in the "native bridge" mode, (i.e. BRIDGE_OPTIMIZATION code in chan_iax2.c). The problem here is that these frames often get to the channel with a different timebase than the system time. Jumps often occur whenever we switch from using "implicit" to "explicit" timestamps and back.  So, I propose a solution to this is the following: 1) add a new member to the channel's pvt structure called "explicit_ts_offset". 2) When we generate timestamps in the "explicit case", do the following: 2a) If pvt->explicit_ts_offset is zero, calculate the difference in time between the present time, and the timestamp on the frame you're about to send.  Store this difference in explicit_ts_offset, and add it from the frame before sending it. 2b) if pvt->explicit_ts_offset is set, add this to the frame we're about to send.  Then, calculate the difference between this time and the current time, and average this into pvt->offset (so that pvt->offset stays synchronized with the explicit ts source). These steps might need to be taken for both the case where frames come from asterisk's core, and the case where "native bridging" is used. I think then we'll end up making nice consistent timestamps. As to why the new jittebuffer is more sensitive to this than the old one, I think it's a combination of issues, including having a longer history period (500 frames vs 50 or so), and other design differences. By: Mark Spencer (markster) 2005-04-01 15:22:52.000-0600 Can you demonstrate a situation in which this timestamp jump occurs? By: zoa (zoa) 2005-04-01 15:34:45.000-0600 try this: iaxclient to cvs-head to cvs-head to pstn delays the audio coming from the PSTN. Please note that all jitter buffers are disabled except for the jitter buffer on the iaxclient. (which cannot be disabled). By: Paul Cadach (pcadach) 2005-04-01 15:39:23.000-0600 For indication about switching from one stream source to another I suggest to use: 1) "mark" bit on RTP; 2) long frames on IAX/IAX2. Using this technique you will know when nternal timestamp stuff should be re-synchronized. By: zoa (zoa) 2005-04-04 03:51:01 i forgot to mention that it does work correctly when having transcoding. Only the native bridging doesnt work. By: Andrew Kohlsmith (akohlsmith) 2005-05-04 10:47:22 Just a datapoint to keep things moving here... A, B = asterisk boxes with new jitter buffer but jitterbuffer=no in iax.conf (DTMF issues).  IAX bridge optimization is not present (patch from C = switch-1.nufone.net with NEWJB *not* defined, IIRC. PRI->A->B->PRI -- no delay issues. PRI->A->B->C->PRI -- the odd 10s one-way-audio problem (frequent enough to be annoying) If I am not mistaken, nufone's network design is actually C being C->D->PRI, with C being a transcode box and D being PRI termination. By: Mark Spencer (markster) 2005-05-04 16:03:49 Can you provide some debug of what happens with the "10 second 1 way audio bug"?  What does it look like on the wire, for example? By: Michael Jerris (mikej) 2005-05-21 09:08:51 Does this issue still exists with current head.  There have been quite a few timestamp fixes that went in recently?? By: Michael Jerris (mikej) 2005-05-25 20:53:10 Okay, one last call.  Is this one resolved in head now? By: Michael Jerris (mikej) 2005-06-06 10:43:04 I have reports that this issue was related to strange settings on some soundcards and was an IAXCOMM specific issue.  If anyone still has this issue with current head, see a bugmarshall in IRC to re-open this issue.