Summary:	ASTERISK-08098: one way audio, when network jitter occur
Reporter:	pj (pj)	Labels:
Date Opened:	2006-11-09 12:04:35.000-0600	Date Closed:	2007-08-26 21:56:34
Priority:	Major	Regression?	No
Status:	Closed/Complete	Components:	Channels/chan_iax2
Versions:		Frequency of Occurrence
Related Issues:
Environment:		Attachments:	( 0) iax.debug1.txt ( 1) iax-debug-02-16.txt
Description:	I'm experiencing call one-way audio after several seconds/minutes. Call itself is not disconnected, but one party stops hear other. Probability of this issue grow, when network jitter grow. Also seems, that when transcoding, one way audio issue probability is higher. It's hard to describe exactly, because this issue appears random, but I can say, that when both call parties (spokes) are connected together via central asterisk (hub), and when both lines are jittery (rtt vary between 100-800ms), probability of one-way audio is maybe 75% in ten minutes call duration! When I look to packed dumps, wireshark shows, that iax minipackets flows in both directions, even in case, that one way audio occurs, nothing special I can see from packet dumps, so I think, that this is internal asterisk problem at higher level. I'm experinecing this issue only in 1.4branch, asterisk 1.2 is working fine. ****** ADDITIONAL INFORMATION ****** I'm using star topology with central asterisk (1.4), that relay iax conections from registered clients (asterisk 1.2), that are behind NAT, so all asterisks have transfer=no, that all trafic going through central asterisk. I read similar issues from asterisk-user mailing list, but seems, that none report this issue in bug tracker. here is my config from central asterisk (hub) for one typical client - asterisk 1.2 (spoke). [honzat]  ; incomming from HT type=user auth=md5 secret=xxx host=dynamic transfer=no jitterbuffer=yes forcejitterbuffer=yes maxjitterbuffer=3000 disallow=all allow=ilbc,gsm,alaw context=from-honzat [honzat-gw] ; outbound to HT type=peer auth=md5 username=abc secret=xxx host=dynamic transfer=no jitterbuffer=yes forcejitterbuffer=yes maxjitterbuffer=3000 qualify=yes qualifysmoothing=yes qualifyfreqok=10000 qualifyfreqnotok=4000 disallow=all allow=ilbc,gsm,alaw context=main
Comments:	By: Joshua C. Colp (jcolp) 2006-11-09 12:47:04.000-0600 An iax2 debug would be great, and if you could turn on debug output to the console and set it to something high like 4 that would also be great. By: pj (pj) 2006-11-09 16:10:26.000-0600 one way audio in this testing (look at iax.debug1.txt) appears about 2m:10s, two client (spokes) are registered (in this example) to central asterisk from one IP (behind one NAT box), all have option transfer=no. By: pj (pj) 2006-11-10 08:29:24.000-0600 additional info: during call, sometimes appears this messages on asterisk console, but I must say, that this doesn't indicate, that issue with one way audio stars immediately after this message. I have 5min successfull call and during this call this notice appears. But before, call had one way audio issue after 1-2mins, and no message appears on console... [Nov 10 15:18:23] NOTICE[24925]: sched.c:283 ast_sched_del: Attempted to delete nonexistent schedule entry 539920! [Nov 10 15:18:54] NOTICE[24920]: sched.c:283 ast_sched_del: Attempted to delete nonexistent schedule entry 543147! By: Tim Panton (mexuar-tim) 2006-11-10 17:01:42.000-0600 Anyone investigating this should also look at http://bugs.digium.com/view.php?id=8273 It seems like the data sent by chan_iax2 sometimes becomes invalid. This bug has voice frames of subclass 138 - which makes no sense to me and 8273 has metaframes that aren't valid. I'm guessng that by the time chan_iax sends a buffer it has be (partially) overwritten. I've spent an hour or so looking at the chan_iax code and can't spot it - but I'm no C coder these days.... By: pj (pj) 2006-11-29 11:30:37.000-0600 still not any idea about this issue? if someone have time to investigate, I'm ready to send any debug what you want ;-) yesterday someone reports another IAX issue, that can also relate with this... 0008432 Asterisk is sending INVAL packages without a reason By: pj (pj) 2006-12-08 08:59:42.000-0600 maybe I found source of the issue, I have folowing setup: UA(skinny)-asteriskA(1.2)---iax(asterisk-central1.4)iax----asteriskB(1.2)-UA(sip) when I disable jitterbuffer on central asterisk (1.4) seems, that this stops issues with one way audio (I have several calls about 10-30mins and are OK). now I have jitterbuffer enabled & forced on iax on asterisk A and B (on both user and peer sections) I will try to upgrade one "branch" asterisk to 1.4 and will report later. By: pj (pj) 2006-12-15 12:17:12.000-0600 I can probably confirm, that asterisk (1.4) have problem when two asterisks are connected using IAX trunk with jitterbuffer enabled on both servers. ie: this setup works (jb means jb enabled & forced): UA(skinny)-asteriskA(1.2/jb)---iax(asterisk-central1.4)iax----asteriskB(1.4/jb)-UA(sip) this make problems with one-way audio, as reported in this tiket UA(skinny)-asteriskA(1.2/jb)---iax/jb(asterisk-central1.4)iax/jb----asteriskB(1.4/jb)-UA(sip) By: Jason Parker (jparker) 2006-12-22 09:48:29.000-0600 Can you verify that this still happens when you remove the skinny endpoint from the mix? By: Paul Belanger (pabelanger) 2006-12-25 17:05:11.000-0600 I'm experience the same issue with one-way audio using IAX to my CO's asterisk box.  I'm using asterisk 1.4, and unsure at the moment what the CO is using. My setup: Polycom 301 --(SIP)-- Asterisk 1.4[trunk] --(IAX)-- CO(Asterisk version unknown) I'll post some IAX2 debug on my next call.  As for my iax.conf, I'm using the default. By: Pedro Tomas (tracinet) 2006-12-29 09:28:27.000-0600 I am actually experiencing the issue with a similar setup, except with all asterisk 1.2.10 servers... In the following scenarios - we have incoming calls delivered to our asteriskA server via a SIP provider which then passes the call to the customer PBX (asteriskB) via IAX which then redirects the caller to another number dialed out of our outbound asterisk server (asteriskC). (jb means jb enabled & forced) the following causes one-way audio, as reported in this ticket: SIP Provider---sip(asteriskA-1.2/jb)---iax(asteriskB-1.2/jb)---iax(asteriskC-1.2/jb)---zap(PRI T-1) this setup seems to work: SIP Provider---sip(asteriskA-1.2/jb)---iax(asteriskB-1.2/jb)---iax(asteriskC-1.2)---zap(PRI T-1) Disabling the jitterbuffer on asteriskC seems to have cleared up the issue for the moment. It is also interesting to note that the following ALWAYS worked: SIP Provider---sip(asteriskA-1.2/jb)---iax(asteriskB-1.2/jb)---iax(asteriskC-1.2/jb)---sip(SIP Provider) This also ALWAYS worked: UA---sip(asteriskB-1.2/jb)---iax(asteriskC-1.2/jb)---zap(PRI T-1) Seems to be isolated to transcoding from IAX to ZAP when more than 2 asterisk servers are in the mix, but not IAX to SIP. Hope this helps....  I can provide DEBUG as needed. By: pj (pj) 2007-01-04 09:40:32.000-0600 yes, the problem still persist, even in case when both endpoints are SIP. (last tested with 1.4 branch (before 1.4 final) >> Can you verify that this still happens when you remove the skinny endpoint from the mix? By: Marcos (mtusso) 2007-01-05 13:31:07.000-0600 I'm also experiencing this problem after updating a server to 1.4 Final.(the 1.4 is also connected to a avaya PBX through a E1) I'm using it to connect to an Asterisk 1.2 server, the problem persists when using Jitterbuffer on 1.4 and without JB also. Same symptoms, after a few seconds/minutes, one way call. Also getting the same Notice event nonexisting schedule entry.... By: pj (pj) 2007-02-11 04:08:23.000-0600 As workaround issues with one-way audio on jittery iax: I disabled jitterbuffer on 'central' asterisk, seems, that this helps. setup with issues: phone(sip)-ast-branchA(jb)===IAX===(jb)central-asterisk(jb)====IAX===(jb)asterisk-branchB-phone(sip) after workaround: phone(sip)-ast-branchA(jb)===IAX===(jb-disable)central-asterisk(jb-disable)====IAX===(jb)asterisk-branchB-phone(sip) seems, that asterisk have problems, when forced iax jitterbuffers are in tandem. central asterisk: SVN-trunk-r53885 branch asterisk: v1.2 By: Tim Panton (mexuar-tim) 2007-02-11 04:16:01.000-0600 Could someone who can reproduce this problem please do a packet capture of a failing call? I'd be happy to dig through an etherreal trace or two. By: Joshua C. Colp (jcolp) 2007-02-15 19:55:47.000-0600 Yes - if anyone is still having this happen with latest version of things please speak up. By: pj (pj) 2007-02-16 06:42:01.000-0600 sad to say, if I enable/force jb on "central/hub" asterisk, after about 10min of talk, one-way audio issue occur. tested on Asterisk SVN-trunk-r54749 uploading iax debug log: iax-debug-02-16.txt By: Tim Panton (mexuar-tim) 2007-02-19 14:24:10.000-0600 Looking at the iax-debug-02-16.txt file the RR_LOSS=33555500  numbers are _mad_ - Part of the problem is display, RR_LOSS can be negative if you get duplicates. Even so you can't lose that many packets :-) By: Serge Vecher (serge-v) 2007-03-07 13:26:14.000-0600 any differences with 1.4.1? By: pj (pj) 2007-03-22 16:03:47 issues with dropped calls/one way audio persist even in current 1.4 branch, SVN-branch-1.4-r59089, sometimes I must stop/start asteriskC (where jb enabled/forced), to be able to successfuly make calls, without this, calls ringing, but after connect I can't hear   remote party. apps like playback, echo, MoH etc is not working until stop/start asteriskC. my setup: phone(sip)-astbranchA(nojb)--iax--alaw(jb)asteriskC(jb)alaw--iax--(nojb)astbranchB-phone(sip) By: Sameer Chunilall (sameerc) 2007-03-28 03:48:17 Having the same issue in an outbound call center environment approx 100 agents running SVN-branch-1.4-r59195. Turn on the jitterbuffer and one way audio will occur, erratically, calls will go silent etc.  tested this many times. We were on 1.2.16 initially and tried SVN-branch-1.4-r59195 - thought that multithreaded IAX would solve our broken audio issues. Reverted back to 1.2.16 and did some load balancing on our 2 servers: our problem seems be sorted, although at the time of writing this, we have been running smooth for only about a half a day. We received the running out of idle threads warning consistently, we played with the default and max thread count setting in iax.conf and the more we increased the thread count, the more the load average climbed. Interestingly though, the audio didn't break up even at the astronomical load averages of 14.5! If we didn't have the one way audio issue, I would probably still be using SVN-branch-1.4-r59195! By: David Chappell (chappell) 2007-04-12 15:47:28 I too experienced this problem between two boxes running 1.4 branch from SVN around April 2nd.  Audio would cut off a minute or two into the call.  It might come back after a minute or two (after most people would have hung up).  I fixed the problem by reverting both boxes to SVN-branch-1.4-r54290. By: pj (pj) 2007-04-16 11:17:37 SVN-branch-1.4-r61658 3 IAX calls (6 call legs), slight jitter, after some minutes, call is droped and audio for _all_ next calls is terrible until asterisk stop/start! By: pj (pj) 2007-05-21 04:10:09 because situation on last weeks slightly changed, I reported my current iax/jb issue in separate bugreport (with iax debug attached), now, I don't observe one-way audio issues. but I must say, that now it is even worst case, because current iax/jb issue efectivelly damage all current and subsequent calls until asterisk is restarted. anyone interesting this bug, please look at: 0009704: jerky all calls until asterisk restart, probably jitterbuffer issue By: pj (pj) 2007-08-06 01:22:25 this issue doesn't longer appear for me, I think you can close this tiket. I think, that was caused because of some deadlock in iax, that was already resolved, as russel wrote in similar issue bugreport: http://bugs.digium.com/view.php?id=9704#64632 By: Russell Bryant (russell) 2007-08-26 21:56:31 I believe that this is fixed in the latest version.  The original reporter indicates that he no longer has the problem.  However, if anyone is still experiencing this, I would like to know so that I can track it down.  Thanks!