| Summary: | ASTERISK-13368: [patch] app_voicemail leaves sockets in close wait. | ||||
| Reporter: | Kevin Stewart (skavin) | Labels: | |||
| Date Opened: | 2009-01-15 15:23:06.000-0600 | Date Closed: | 2009-04-15 17:07:43 | ||
| Priority: | Minor | Regression? | No | ||
| Status: | Closed/Complete | Components: | Applications/app_voicemail/IMAP | ||
| Versions: | Frequency of Occurrence | ||||
| Related Issues: |
| ||||
| Environment: | Attachments: | ( 0) 20090218__bug14253.diff.txt ( 1) 20090219__bug14253.diff.txt ( 2) crash-bt.txt ( 3) openfiles-week.JPG ( 4) valgrind.txt | |||
| Description: | app_voicemail is not closing imap sockets and they are going into CLOSE_WAIT. I am using imap-2006j open files seem to go up during busy times but never drop. my asterisk-vm process does not notify it is only there to isolate recording and playback of voice mail from the rest of asterisk. my asterisk process forwards to asterisk-vm in the case of busy/noanswer or checking voicemail. in 50 hours I have generated 836 handles in CLOSE_WAIT all to imap 33 handles in ESTABLISHED 1 mysql 32 imap 12 udp handles | ||||
| Comments: | By: Kevin Stewart (skavin) 2009-01-15 15:30:55.000-0600 I have attached a graph to show growth of file handles over time. the sudden drops are cashes of the voice mail process which I am tracking down and will log later but at least they mask the selectable socket depletion. By: Mark Michelson (mmichelson) 2009-01-15 17:50:38.000-0600 Hmm, interesting. When I see this, I'm really not sure that it is something that is the fault of Asterisk or the c-client. The c-client handles all socket-level operations, so it is the job of the c-client to close any sockets that need closing. If the c-client is waiting for some input from Asterisk prior to closing the open connections, then I certainly don't know what it wants. By: Tilghman Lesher (tilghman) 2009-01-29 15:54:48.000-0600 I don't pretend to know much about IMAP, but let's try this patch and see if it makes any difference. By: Kevin Stewart (skavin) 2009-02-01 20:09:36.000-0600 Am testing the patch and will give you a report in an hour or so By: Kevin Stewart (skavin) 2009-02-01 22:31:44.000-0600 I applied the patch vs 1.4.21.2 I am still getting sockets in close wait. and I can now crash asterisk in mail_open_work in mail.c with (char *)stream being text not the stream struct. so it looks like stream is being used after it is freed I was leaving a message and checked voice mail while that recording was still active causing the crash By: Tilghman Lesher (tilghman) 2009-02-02 14:15:50.000-0600 Okay, you reported a crash. I'd like to see a stack backtrace from that crash, please. By: Kevin Stewart (skavin) 2009-02-02 14:24:52.000-0600 I have attached the back trace By: Tilghman Lesher (tilghman) 2009-02-02 15:30:47.000-0600 Okay, on further examination, it appears that the IMAP support in app_voicemail is one huge resource leak, which cannot be easily remedied. It engages in sharing of resources, migration of multiple pointers which are copied between structures, etc. It's clear from the implementation that this was done for the purpose of making IMAP support very fast. Unfortunately, the code also leaks like a sieve. In fact, the way it's written, I'm surprised that it doesn't constantly blow up. By: Tilghman Lesher (tilghman) 2009-02-04 17:22:14.000-0600 After reviewing it further, I think I may be able to patch this up a bit. See attached. By: Kevin Stewart (skavin) 2009-02-07 03:13:48.000-0600 Am working on getting this into testing its a bigger patch so must jump through more hoops on my side. By: Kevin Stewart (skavin) 2009-02-08 22:50:06.000-0600 tryed to apply your patch to 1.4 and got this error [CC] app_voicemail.c -> app_voicemail.o In file included from /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/osdep.h:63, from /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/c-client.h:42, from app_voicemail.c:82: /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/env_unix.h:71: warning: function declaration isn’t a prototype app_voicemail.c:126: warning: data definition has no type or storage class app_voicemail.c:126: warning: type defaults to ‘int’ in declaration of ‘AST_THREADSTORAGE’ app_voicemail.c:126: warning: parameter names (without types) in function declaration app_voicemail.c: In function ‘create_vm_state_from_user’: app_voicemail.c:1902: error: ‘ts_vmstate’ undeclared (first use in this function) app_voicemail.c:1902: error: (Each undeclared identifier is reported only once app_voicemail.c:1902: error: for each function it appears in.) app_voicemail.c: In function ‘get_vm_state_by_imapuser’: app_voicemail.c:1928: error: ‘ts_vmstate’ undeclared (first use in this function) app_voicemail.c: In function ‘get_vm_state_by_mailbox’: app_voicemail.c:1968: error: ‘ts_vmstate’ undeclared (first use in this function) app_voicemail.c: In function ‘vm_execmain’: app_voicemail.c:7350: error: ‘ts_vmstate’ undeclared (first use in this function) make[1]: *** [app_voicemail.o] Error 1 make[1]: Leaving directory `/usr/src/svn/asterisk-svn-1.4.23.1/apps' make: *** [apps] Error 2 added #include "asterisk/threadstorage.h" and got this error In file included from /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/osdep.h:63, from /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/c-client.h:42, from app_voicemail.c:82: /usr/src/ray/asterisk-1.4.21.2/imap-2007e/c-client/env_unix.h:71: warning: function declaration isn’t a prototype app_voicemail.c:127:29: error: macro "AST_THREADSTORAGE" requires 2 arguments, but only 1 given app_voicemail.c:127: warning: data definition has no type or storage class app_voicemail.c:127: warning: type defaults to ‘int’ in declaration of ‘AST_THREADSTORAGE’ app_voicemail.c: In function ‘create_vm_state_from_user’: app_voicemail.c:1903: error: ‘ts_vmstate’ undeclared (first use in this function) app_voicemail.c:1903: error: (Each undeclared identifier is reported only once app_voicemail.c:1903: error: for each function it appears in.) thoughts? should there be a ts_vmstate_init? By: Tilghman Lesher (tilghman) 2009-02-09 11:40:03.000-0600 Whoops. Patch updated. By: Kevin Stewart (skavin) 2009-02-17 16:11:43.000-0600 crash on voice mail access hangup so *55 ... you have no new voicemails <hangup>crash (gdb) bt #0 messagecount (context=0xb6e1aa33 "default", mailbox=0xb6e1aa28 "6499744238", folder=0x0) at app_voicemail.c:1204 #1 0xb709d683 in has_voicemail (mailbox=0xb6e22b7d "6499744238@default", folder=0x0) at app_voicemail.c:1425 #2 0xb70b0052 in vm_execmain (chan=0x83379c0, data=0xb6e24f28) at app_voicemail.c:7759 #3 0x080ceaf7 in pbx_extension_helper (c=0x83379c0, con=0x0, context=0x8337b40 "default", exten=0x8337b90 "vmac6499744238", priority=3, label=0x0, callerid=0x83332a8 "6499744238", action=E_SPAWN) at pbx.c:537 #4 0x080d0641 in __ast_pbx_run (c=0x83379c0) at pbx.c:2320 ASTERISK-1 0x080d16ce in pbx_thread (data=0x83379c0) at pbx.c:2627 ASTERISK-2 0x08100e70 in dummy_start (data=0x8333408) at utils.c:856 ASTERISK-3 0xb7f434c0 in start_thread () from /lib/i686/cmov/libpthread.so.0 ASTERISK-4 0xb7e556de in clone () from /lib/i686/cmov/libc.so.6 (gdb) print vms_p $1 = (struct vm_state *) 0x4f424e49 (gdb) print (char*)vms_p $2 = 0x4f424e49 <Address 0x4f424e49 out of bounds> By: Tilghman Lesher (tilghman) 2009-02-18 17:12:33.000-0600 I wonder if the valgrind output might be helpful, in this case. By: Tilghman Lesher (tilghman) 2009-02-18 19:42:19.000-0600 Aha, found it. New patch uploaded (for 1.4). By: Kevin Stewart (skavin) 2009-02-19 15:38:00.000-0600 no crash on hangup of voice mail access with new patch no crash on multiple access of voice mail 1 leaving and one picking up voice mail for same account at the same time. I am getting errors on the console -- <SIP/119.224.129.2-08326648> Playing 'vm-theperson' (language 'en') -- <SIP/119.224.129.2-08326648> Playing 'digits/3' (language 'en') -- <SIP/119.224.129.2-08326648> Playing 'digits/8' (language 'en') -- <SIP/119.224.129.2-08326648> Playing 'vm-isunavail' (language 'en') [Feb 20 10:34:06] WARNING[30100]: app_voicemail.c:1750 mm_log: IMAP Warning: Unknown message data: 1 FETCH [Feb 20 10:34:06] WARNING[30100]: app_voicemail.c:1750 mm_log: IMAP Warning: Unknown message data: 2 FETCH [Feb 20 10:34:06] ERROR[30100]: app_voicemail.c:1625 mm_searched: No state found. [Feb 20 10:34:06] ERROR[30100]: app_voicemail.c:1625 mm_searched: No state found. [Feb 20 10:34:06] WARNING[30100]: app_voicemail.c:1750 mm_log: IMAP Warning: Bad quota resource list for ROOT -- <SIP/119.224.129.2-08326648> Playing 'beep' (language 'en') -- Recording the message -- x=0, open writing: /var/spool/asterisk/voicemail/default/38/tmp/x0Ocj4 format: wav, 0x8341288 -- x=1, open writing: /var/spool/asterisk-vm/voicemail/default/38/tmp/x0Ocj4 format: gsm, 0x83414c8 -- User hung up [Feb 20 10:34:10] ERROR[30100]: app_voicemail.c:1625 mm_searched: No state found. [Feb 20 10:34:10] ERROR[30100]: app_voicemail.c:1625 mm_searched: No state found. [Feb 20 10:34:10] ERROR[30100]: app_voicemail.c:1625 mm_searched: No state found. == Spawn extension (default, 38, 3) exited non-zero on 'SIP/119.224.129.2-08326648' -- Executing [h@default:1] GotoIf("SIP/119.224.129.2-08326648", "?gresyncvm:gnoresyncvm") in new stack -- Goto (default,h,5) -- Executing [h@default:5] Hangup("SIP/119.224.129.2-08326648", "") in new stack By: Tilghman Lesher (tilghman) 2009-02-19 15:50:55.000-0600 Patch updated that should take care of the errors. That may take care of the warnings, too. By: Digium Subversion (svnbot) 2009-02-19 16:26:02.000-0600 Repository: asterisk Revision: 177536 U branches/1.4/apps/app_voicemail.c ------------------------------------------------------------------------ r177536 | tilghman | 2009-02-19 16:26:02 -0600 (Thu, 19 Feb 2009) | 7 lines Fix up potential crashes, by reducing the sharing between interactive and non-interactive threads. (closes issue ASTERISK-13368) Reported by: Skavin Patches: 20090219__bug14253.diff.txt uploaded by Corydon76 (license 14) Tested by: Skavin ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=177536 By: Digium Subversion (svnbot) 2009-02-19 16:33:01.000-0600 Repository: asterisk Revision: 177537 _U trunk/ U trunk/apps/app_voicemail.c ------------------------------------------------------------------------ r177537 | tilghman | 2009-02-19 16:33:00 -0600 (Thu, 19 Feb 2009) | 14 lines Merged revisions 177536 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r177536 | tilghman | 2009-02-19 16:26:01 -0600 (Thu, 19 Feb 2009) | 7 lines Fix up potential crashes, by reducing the sharing between interactive and non-interactive threads. (closes issue ASTERISK-13368) Reported by: Skavin Patches: 20090219__bug14253.diff.txt uploaded by Corydon76 (license 14) Tested by: Skavin ........ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=177537 By: Digium Subversion (svnbot) 2009-02-19 16:34:37.000-0600 Repository: asterisk Revision: 177538 _U branches/1.6.0/ U branches/1.6.0/apps/app_voicemail.c ------------------------------------------------------------------------ r177538 | tilghman | 2009-02-19 16:34:37 -0600 (Thu, 19 Feb 2009) | 21 lines Merged revisions 177537 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r177537 | tilghman | 2009-02-19 16:33:00 -0600 (Thu, 19 Feb 2009) | 14 lines Merged revisions 177536 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r177536 | tilghman | 2009-02-19 16:26:01 -0600 (Thu, 19 Feb 2009) | 7 lines Fix up potential crashes, by reducing the sharing between interactive and non-interactive threads. (closes issue ASTERISK-13368) Reported by: Skavin Patches: 20090219__bug14253.diff.txt uploaded by Corydon76 (license 14) Tested by: Skavin ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=177538 By: Digium Subversion (svnbot) 2009-02-19 16:35:48.000-0600 Repository: asterisk Revision: 177539 _U branches/1.6.1/ U branches/1.6.1/apps/app_voicemail.c ------------------------------------------------------------------------ r177539 | tilghman | 2009-02-19 16:35:48 -0600 (Thu, 19 Feb 2009) | 21 lines Merged revisions 177537 via svnmerge from https://origsvn.digium.com/svn/asterisk/trunk ................ r177537 | tilghman | 2009-02-19 16:33:00 -0600 (Thu, 19 Feb 2009) | 14 lines Merged revisions 177536 via svnmerge from https://origsvn.digium.com/svn/asterisk/branches/1.4 ........ r177536 | tilghman | 2009-02-19 16:26:01 -0600 (Thu, 19 Feb 2009) | 7 lines Fix up potential crashes, by reducing the sharing between interactive and non-interactive threads. (closes issue ASTERISK-13368) Reported by: Skavin Patches: 20090219__bug14253.diff.txt uploaded by Corydon76 (license 14) Tested by: Skavin ........ ................ ------------------------------------------------------------------------ http://svn.digium.com/view/asterisk?view=rev&revision=177539 | ||||
