Summary: | ASTERISK-09640: IAX peers becomes unreachable/reachable | ||
Reporter: | Thiago Garcia (thiagarcia) | Labels: | |
Date Opened: | 2007-06-10 19:29:36 | Date Closed: | 2007-07-06 19:18:18 |
Priority: | Major | Regression? | No |
Status: | Closed/Complete | Components: | Core/General |
Versions: | Frequency of Occurrence | ||
Related Issues: | |||
Environment: | Attachments: | ( 0) iax2debug.zip | |
Description: | My IAX peers becomes unreachable/reachable. The issue appeared after upgrade SVN from branches 1.4 67158 to 68527. I am using softphone idefisk 1.31 for linux and codec GSM. _______________________________________________________________________________ [Jun 10 21:00:15] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.65 (callno 56763) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:00:35] NOTICE[18375]: chan_iax2.c:7211 socket_process: Peer '5644' is now REACHABLE! Time: 8 [Jun 10 21:00:55] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34180) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:00:59] NOTICE[18397]: chan_iax2.c:8005 __iax2_poke_noanswer: Peer '5628' is now UNREACHABLE! Time: 1 [Jun 10 21:01:09] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34182) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:01:29] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34183) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:01:35] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.65 (callno 56766) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:01:39] NOTICE[18424]: chan_iax2.c:8005 __iax2_poke_noanswer: Peer '5644' is now UNREACHABLE! Time: 8 [Jun 10 21:01:49] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34184) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:01:49] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.65 (callno 56768) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:02:09] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34186) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:02:09] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.65 (callno 56769) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:02:29] WARNING[18757]: chan_iax2.c:6322 socket_read: Dropping frame from 10.9.15.181 (callno 34187) of type 6 (subclass 3) due to frame of type 6 (subclass 4) already in process [Jun 10 21:02:29] NOTICE[18478]: chan_iax2.c:7211 socket_process: Peer '5644' is now REACHABLE! Time: 4 ______________________________________________________________________________ If i revert chan_iax2.c to svn 67270 branches 1.4 the following error occurs. [Jun 10 21:13:22] WARNING[21941]: chan_iax2.c:6318 socket_read: Dropping full frame from 10.9.15.65 (callno 56804) received too rapidly [Jun 10 21:13:26] NOTICE[21496]: chan_iax2.c:7997 __iax2_poke_noanswer: Peer '5644' is now UNREACHABLE! Time: 10 [Jun 10 21:13:36] NOTICE[21499]: chan_iax2.c:7203 socket_process: Peer '5644' is now REACHABLE! Time: 8 _________________________________________________________________________ If revert the chan_iax2.c to svn 67158, the issue does not occur. | ||
Comments: | By: Eliel Sardanons (eliel) 2007-06-10 20:34:24 Is this issue a duplicate of bug ASTERISK-9394 ? By: Thiago Garcia (thiagarcia) 2007-06-10 21:26:17 The bug ASTERISK-9394 relates that when restarts the service come back to the normal, for me the problem always occurs. By: pj (pj) 2007-06-11 03:27:44 is working fine, for me SVN-trunk-r68575M chan_iax2.c Revision: 68453 idefisk 1.37/wxp By: Fabricio Machado (fmachado) 2007-06-15 14:55:16 I have same problem. It occurs only on idefisk linux version 1.31. By: Russell Bryant (russell) 2007-06-15 21:47:06 This should already be fixed from revision 69012. Please reopen if you still have a problem with the latest code in the 1.4 branch, or 1.4.5. By: Thiago Garcia (thiagarcia) 2007-06-20 08:43:16 The issue still occurs in my dual-processor server with any softphones. [Jun 20 10:37:31] NOTICE[4868]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5690' is now UNREACHABLE! Time: 2 [Jun 20 10:37:35] NOTICE[4862]: chan_iax2.c:7221 socket_process: Peer '5668' is now REACHABLE! Time: 7 [Jun 20 10:37:41] NOTICE[4868]: chan_iax2.c:7221 socket_process: Peer '5690' is now REACHABLE! Time: 3 [Jun 20 10:37:51] NOTICE[4868]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5643' is now UNREACHABLE! Time: 3 [Jun 20 10:37:51] NOTICE[4865]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5545' is now UNREACHABLE! Time: 8 [Jun 20 10:38:05] NOTICE[4866]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5648' is now UNREACHABLE! Time: 8 [Jun 20 10:38:05] NOTICE[4869]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5638' is now UNREACHABLE! Time: 6 [Jun 20 10:38:05] NOTICE[4863]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5639' is now UNREACHABLE! Time: 2 [Jun 20 10:38:05] NOTICE[4861]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5547' is now UNREACHABLE! Time: 11 [Jun 20 10:38:05] NOTICE[4862]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5674' is now UNREACHABLE! Time: 10 [Jun 20 10:38:07] NOTICE[4863]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5632' is now UNREACHABLE! Time: 13 [Jun 20 10:38:15] NOTICE[4867]: chan_iax2.c:7221 socket_process: Peer '5648' is now REACHABLE! Time: 1 [Jun 20 10:38:15] NOTICE[4860]: chan_iax2.c:7221 socket_process: Peer '5639' is now REACHABLE! Time: 3 [Jun 20 10:38:15] NOTICE[4868]: chan_iax2.c:7221 socket_process: Peer '5674' is now REACHABLE! Time: 13 [Jun 20 10:38:19] NOTICE[4866]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5534' is now UNREACHABLE! Time: 2 [Jun 20 10:38:21] NOTICE[4863]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5651' is now UNREACHABLE! Time: 12 [Jun 20 10:38:21] NOTICE[4868]: chan_iax2.c:7221 socket_process: Peer '5643' is now REACHABLE! Time: 7 [Jun 20 10:38:25] NOTICE[4867]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5614' is now UNREACHABLE! Time: 4 [Jun 20 10:38:26] NOTICE[4869]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5625' is now UNREACHABLE! Time: 14 [Jun 20 10:38:29] NOTICE[4861]: chan_iax2.c:7221 socket_process: Peer '5534' is now REACHABLE! Time: 4 [Jun 20 10:38:31] NOTICE[4863]: chan_iax2.c:7221 socket_process: Peer '5651' is now REACHABLE! Time: 9 [Jun 20 10:38:35] NOTICE[4860]: chan_iax2.c:7221 socket_process: Peer '5614' is now REACHABLE! Time: 13 [Jun 20 10:38:35] NOTICE[4869]: chan_iax2.c:7221 socket_process: Peer '5638' is now REACHABLE! Time: 13 [Jun 20 10:38:36] NOTICE[4867]: chan_iax2.c:7221 socket_process: Peer '5625' is now REACHABLE! Time: 8 [Jun 20 10:38:37] NOTICE[4865]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5635' is now UNREACHABLE! Time: 9 [Jun 20 10:38:39] NOTICE[4869]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5668' is now UNREACHABLE! Time: 7 [Jun 20 10:38:41] NOTICE[4862]: chan_iax2.c:7221 socket_process: Peer '5545' is now REACHABLE! Time: 1 _________________________________________________________________________________ [Jun 20 11:04:28] NOTICE[4863]: chan_iax2.c:7221 socket_process: Peer '5547' is now REACHABLE! Time: 5 [Jun 20 11:04:32] NOTICE[4866]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5668' is now UNREACHABLE! Time: 13 [Jun 20 11:04:33] NOTICE[4860]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5610' is now UNREACHABLE! Time: 3 [Jun 20 11:04:34] NOTICE[4864]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5544' is now UNREACHABLE! Time: 3 [Jun 20 11:04:34] NOTICE[4866]: chan_iax2.c:7221 socket_process: Peer '5545' is now REACHABLE! Time: 17 [Jun 20 11:04:34] NOTICE[4864]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5632' is now UNREACHABLE! Time: 5 [Jun 20 11:04:35] NOTICE[4869]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5553' is now UNREACHABLE! Time: 3 [Jun 20 11:04:37] NOTICE[4869]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5555' is now UNREACHABLE! Time: 4 [Jun 20 11:04:38] NOTICE[4861]: chan_iax2.c:7221 socket_process: Peer '5622' is now REACHABLE! Time: 11 [Jun 20 11:04:39] WARNING[11524]: app_dial.c:741 wait_for_answer: Unable to forward voice or dtmf [Jun 20 11:04:39] NOTICE[4862]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5651' is now UNREACHABLE! Time: 13 pbx-fns*CLI> iax2 show channels Channel Peer Username ID (Lo/Rem) Seq (Tx/Rx) Lag Jitter JitBuf Format IAX2/5614-4 10.9.15.31 (None) 00004/02582 00177/00189 00083ms 0127ms 0201ms ulaw (None) 10.9.15.195 (None) 00005/07831 00001/00000 00000ms 0000ms 0040ms unknow (None) 10.9.15.63 (None) 00042/13632 00001/00000 00000ms 0000ms 0040ms unknow (None) 10.9.15.99 (None) 00044/03959 00001/00000 00000ms 0000ms 0040ms unknow (None) 10.9.15.157 (None) 00049/29062 00001/00000 00000ms 0000ms 0040ms unknow (None) 10.9.15.64 (None) 00054/19679 00001/00000 00000ms 0000ms 0040ms unknow IAX2/5661-61 10.9.15.23 5661 00061/30033 00049/00048 00118ms 0000ms 0040ms gsm IAX2/5539-69 10.9.15.157 5539 00069/29063 00004/00003 00000ms 0064ms 0104ms gsm (None) 10.9.15.31 (None) 00074/02608 00001/00000 00000ms 0000ms 0040ms unknow (None) 10.9.15.193 (None) 00078/29222 00001/00000 00000ms 0000ms 0040ms unknow IAX2/5628-79 10.9.15.181 (None) 00079/01748 00045/00049 00092ms 0000ms 0040ms gsm IAX2/5645-123 10.9.15.123 5645 00123/22592 00167/00188 00079ms 0080ms 0135ms gsm (None) 10.9.15.80 (None) 00178/21799 00001/00000 00000ms 0000ms 0040ms unknow IAX2/5649-179 10.9.15.164 5649 00179/30166 00006/00004 00000ms 0112ms 0192ms gsm IAX2/5669-180 10.9.15.106 (None) 00180/28768 00002/00003 00000ms 0000ms 0040ms unknow 15 active IAX channels [Jun 20 11:04:42] NOTICE[4863]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5614' is now UNREACHABLE! Time: 21 [Jun 20 11:04:42] NOTICE[4865]: chan_iax2.c:7221 socket_process: Peer '5668' is now REACHABLE! Time: 3 [Jun 20 11:04:42] NOTICE[4862]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5654' is now UNREACHABLE! Time: 13 [Jun 20 11:04:44] NOTICE[4863]: chan_iax2.c:7221 socket_process: Peer '5544' is now REACHABLE! Time: 19 [Jun 20 11:04:44] NOTICE[4861]: chan_iax2.c:7221 socket_process: Peer '5632' is now REACHABLE! Time: 10 ________________________________________________________________________ pbx-fns*CLI> iax2 show stats IAX Statistics --------------------- Outstanding frames: 43 (33 ingress, 10 egress) Packets in transmit queue: 10 dead, 2 final, 10 total _____________________________________________________________________________ pbx-fns*CLI> iax2 show channels Channel Peer Username ID (Lo/Rem) Seq (Tx/Rx) Lag Jitter JitBuf Format (None) 10.9.15.98 (None) 00131/21423 00001/00000 00000ms 0000ms 0040ms unknow 1 active IAX channel [Jun 20 12:48:33] NOTICE[24152]: chan_iax2.c:8023 __iax2_poke_noanswer: Peer '5660' is now UNREACHABLE! Time: 14 By: Russell Bryant (russell) 2007-06-20 13:38:07 Can you get an iax2 debug trace or a wireshark capture of the IAX2 traffic where you are seeing this happen? By: Thiago Garcia (thiagarcia) 2007-06-20 14:39:17 iax2 debug trace attached in iax2debug.zip. By: Russell Bryant (russell) 2007-06-21 13:43:18 Alright, I think I know what the issue is here. I have created a branch for working on some changes that should fix it. I will report progress here. http://svn.digium.com/svn/asterisk/team/russell/iax2_ff By: Russell Bryant (russell) 2007-06-21 16:19:14 I think I have this fixed in 1.4 and trunk in revisions 70866 and 70877. Let me know if you still have problems. Thank you! By: Arcadiy Ivanov (arcivanov) 2007-06-21 21:43:20 I have to say I experience the same - peers don't register, all full packets are deemed corrupted. By: Arcadiy Ivanov (arcivanov) 2007-06-21 21:50:10 BTW: This patch is in 1.4 branch (don't know about trunk). Can someone roll it back in 1.4 until the issue is resolved? By: Russell Bryant (russell) 2007-06-21 22:15:30 I apologize for the breakage. I made a small typo which completely broke chan_iax2. Anyway, it should be good now in 1.4 and trunk in revisions 71003 and 71004. Thanks! By: Thiago Garcia (thiagarcia) 2007-06-27 13:47:06 Do you need any other information ? How can we help you ? This issue is messing up my queues. By: Russell Bryant (russell) 2007-06-27 14:34:17 It is really hard to tell, but from the debug output you provided, it doesn't look like a bug in Asterisk. It looks like a bug in iaxclient. If you want to get a better packet trace so that we can determine exactly, then use wireshark, and capture for a longer period of time. Then, others can open it with wireshark and analyze it. In your debug output, here is all of the traffic to/from 10.9.15.156: Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: POKE Timestamp: 00003ms SCall: 00032 DCall: 00000 [10.9.15.156:4569] Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: ACK Timestamp: 00003ms SCall: 02047 DCall: 00032 [10.9.15.156:4569] [Jun 22 19:25:04] NOTICE[1865]: chan_iax2.c:8101 __iax2_poke_noanswer: Peer '5689' is now UNREACHABLE! Time: 10 Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: REGREQ Timestamp: 00003ms SCall: 02048 DCall: 00000 [10.9.15.156:4569] USERNAME : 5689 REFRESH : 60 Tx-Frame Retry[-01] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: ACK Timestamp: 00003ms SCall: 00010 DCall: 02048 [10.9.15.156:4569] Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: REGACK Timestamp: 00018ms SCall: 00010 DCall: 02048 [10.9.15.156:4569] USERNAME : 5689ug off DATE TIME : 2007-06-22 19:25:12 REFRESH : 60ebug off APPARENT ADDRES : IPV4 10.9.15.156:4569 pbx-fns*CLI> iax2 set debug off Rx-Frame Retry[ No] -- OSeqno: 001 ISeqno: 001 Type: IAX Subclass: ACK Timestamp: 00018ms SCall: 02048 DCall: 00010 [10.9.15.156:4569] Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: PONG Timestamp: 00006ms SCall: 06506 DCall: 00008 [10.9.15.156:4569] RR_JITTER : 0 RR_LOSS : 0 RR_PKTS : 1 RR_DELAY : 40 RR_DROPPED : 0 RR_OUTOFORDER : 0 Tx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: INVAL Timestamp: 00000ms SCall: 00008 DCall: 06506 [10.9.15.156:4569] Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: POKE Timestamp: 00008ms SCall: 00012 DCall: 00000 [10.9.15.156:4569] Tx-Frame Retry[000] -- OSeqno: 000 ISeqno: 000 Type: IAX Subclass: POKE Timestamp: 00016ms SCall: 00028 DCall: 00000 [10.9.15.65:4569] Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: ACK Timestamp: 00008ms SCall: 02049 DCall: 00012 [10.9.15.156:4569] Rx-Frame Retry[ No] -- OSeqno: 000 ISeqno: 001 Type: IAX Subclass: ACK Timestamp: 00016ms SCall: 05541 DCall: 00028 [10.9.15.65:4569] After every POKE, there should eventually be a PONG. However, the only PONG shown in this output is one that is in response to nothing shown here. Also, Asterisk doesn't know what it is related to, either, because it responds with an INVAL. By: Russell Bryant (russell) 2007-06-29 09:48:11 Also note that it is not necessarily a bug in iaxclient, as it could simply be network problems. Are these clients on a local network, or is there a greater possibility that they could be losing packets? By: Thiago Garcia (thiagarcia) 2007-06-29 10:34:58 If i revert the chan_iax2.c to the SVN 67158 the problem does not occur. All of the clients are in a network 100baseTx-FD, this is output of ifconfig in my PBX. RX packets:38705055 errors:0 dropped:0 overruns:0 frame:0 TX packets:40030031 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 The network do not have problems detected. ping -s 1500 pbx-fns.xxxxxx.com 146 packets transmitted, 146 received, 0% packet loss, time 145033ms rtt min/avg/max/mdev = 1.131/1.202/1.271/0.043 ms By: Thiago Garcia (thiagarcia) 2007-07-03 08:28:45 The issue occurs in "idefisk 1.31 Linux", "idefisk 1.37 winxp". In "idefisk 2.0 winxp" not occurs, I send email to support@asteriskguru.com, but did not they answer me. Exists information that can pass for support asteriskguru, about this modification? ______________________________________________________________________________ Revision 67270 - (view) (download) (as text) (annotate) - [select for diffs] Modified Tue Jun 5 14:35:52 2007 UTC (3 weeks, 6 days ago) by kpfleming File length: 337259 byte(s) Diff to previous 67158 ensure that a burst of full frames (AST_FRAME_DTMF being the prime example) will not be processed out of order... this is a brute force fix, but seems to be the safest fix for now (thanks to the Digium PQ department for finding this bug) _______________________________________________________________________________ If i revert asterisk to branches SVN 67158, the issue don't occurs. By: Russell Bryant (russell) 2007-07-03 09:57:18 Can you verify that you tried the 1.4 branch after revision 71004? It is not clear from the history of this report. By: Thiago Garcia (thiagarcia) 2007-07-03 10:02:11 Yes. Now i am running SVN-branch-1.4-r72933. By: Russell Bryant (russell) 2007-07-05 11:27:40 Good news! I have finally been able to reproduce this locally. I will keep working on it and post details once I am able to get more details on the problem. By: Digium Subversion (svnbot) 2007-07-05 17:48:56 Repository: asterisk Revision: 73555 ------------------------------------------------------------------------ r73555 | russell | 2007-07-05 17:48:55 -0500 (Thu, 05 Jul 2007) | 3 lines copy from the correct buffer when deferring a full frame (related to issue ASTERISK-9640) ------------------------------------------------------------------------ By: Digium Subversion (svnbot) 2007-07-05 17:49:23 Repository: asterisk Revision: 73557 ------------------------------------------------------------------------ r73557 | russell | 2007-07-05 17:49:22 -0500 (Thu, 05 Jul 2007) | 11 lines Merged revisions 73555 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r73555 | russell | 2007-07-05 18:05:33 -0500 (Thu, 05 Jul 2007) | 3 lines copy from the correct buffer when deferring a full frame (related to issue ASTERISK-9640) ........ ------------------------------------------------------------------------ By: Digium Subversion (svnbot) 2007-07-05 17:50:04 Repository: asterisk Revision: 73551 ------------------------------------------------------------------------ r73551 | russell | 2007-07-05 17:31:31 -0500 (Thu, 05 Jul 2007) | 7 lines * Store the call number that a thread is processing without the full frame bit set to ease debugging * When deferring a full frame for processing, stick it into the queue for the thread that is processing frames for that call, not the one that read the current frame and is about to go back into the idle list (related to issue ASTERISK-9640) ------------------------------------------------------------------------ By: Digium Subversion (svnbot) 2007-07-05 17:50:19 Repository: asterisk Revision: 73552 ------------------------------------------------------------------------ r73552 | russell | 2007-07-05 17:32:30 -0500 (Thu, 05 Jul 2007) | 15 lines Merged revisions 73551 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r73551 | russell | 2007-07-05 17:31:31 -0500 (Thu, 05 Jul 2007) | 6 lines * Store the call number that a thread is processing without the full frame bit set to ease debugging * When deferring a full frame for processing, stick it into the queue for the thread that is processing frames for that call, not the one that read the current frame and is about to go back into the idle list (related to issue ASTERISK-9640) ........ ------------------------------------------------------------------------ By: Russell Bryant (russell) 2007-07-05 17:55:11 Alright, give it another try after the commits that I made today. By: Thiago Garcia (thiagarcia) 2007-07-06 09:32:17 I upgrade yesterday, and until now, there is no issues. I am monitoring today and if there is another issue, i will report it. Thanks for the attention, and sorry for my English. By: Russell Bryant (russell) 2007-07-06 13:51:54 I'm closing this out as I think I finally have this fixed. Please let me know if you have any more problems. |