original command line = asterisk -vvvvvfdg Asterisk CVS-01/01/04-19:50:33 Hardware =>Dual 2.4G Xeon (w/hyperthreading enabled) 2G Mem All calls are inbound SIP routed outbound as SIP from 2 VoIP gateways. TestBed Configuration --------------------- ----------------------------------------- | | | Abacus Call Generator | | | |_________________________________________ | | | 7 Inbound T1's | 7 Outbound T1's | | -------- -------- | VoIP | | VoIP | | GW 2 | | GW 1 | |________| |________| | | | Inbound SIP Calls | | | | --------------------- | |_____ | 100MBps |_____| Outbound SIP Calls | Ethernet Switch | |_____________________| | | --------------------- | | | Asterisk | |_____________________| Calls were generated a rate of 1 new call per second. With a call duration of 180 seconds and a 2 second pause between hanging up and starting a new call. Test ran for 12 hours and 17 minutes before segfaulting. All calls were G711. sip.conf -------- [general] port = 5060 bindaddr = 10.31.4.60 context = default dtmfmode=inband allow=ulaw allow=gsm extensions.conf --------------- [general] static=yes [default] exten => _555.,1,Dial(SIP/${EXTEN}@10.31.15.253:4000) ; exten => 0000,1,VoiceMailMain(); ########################################### Begin debugging session ############################################## GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... Core was generated by `asterisk -vvvvvfdg'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libpthread.so.0...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libncurses.so.5...done. Loaded symbols for /lib/libncurses.so.5 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/asterisk/modules/chan_modem.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_modem.so Reading symbols from /usr/lib/asterisk/modules/chan_modem_aopen.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_modem_aopen.so Reading symbols from /usr/lib/asterisk/modules/res_adsi.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_adsi.so Reading symbols from /usr/lib/asterisk/modules/res_parking.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_parking.so Reading symbols from /usr/lib/asterisk/modules/res_crypto.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_crypto.so Reading symbols from /usr/lib/libssl.so.0.9.6...done. Loaded symbols for /usr/lib/libssl.so.0.9.6 Reading symbols from /usr/lib/libcrypto.so.0.9.6...done. Loaded symbols for /usr/lib/libcrypto.so.0.9.6 ---Type to continue, or q to quit--- Reading symbols from /usr/lib/asterisk/modules/res_musiconhold.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_musiconhold.so Reading symbols from /usr/lib/asterisk/modules/res_indications.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_indications.so Reading symbols from /usr/lib/asterisk/modules/res_monitor.so...done. Loaded symbols for /usr/lib/asterisk/modules/res_monitor.so Reading symbols from /usr/lib/asterisk/modules/chan_iax.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_iax.so Reading symbols from /usr/lib/asterisk/modules/chan_sip.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_sip.so Reading symbols from /usr/lib/asterisk/modules/chan_modem_bestdata.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_modem_bestdata.so Reading symbols from /usr/lib/asterisk/modules/chan_modem_i4l.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_modem_i4l.so Reading symbols from /usr/lib/asterisk/modules/chan_agent.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_agent.so Reading symbols from /usr/lib/asterisk/modules/chan_mgcp.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_mgcp.so Reading symbols from /usr/lib/asterisk/modules/chan_iax2.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_iax2.so Reading symbols from /usr/lib/asterisk/modules/chan_local.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_local.so Reading symbols from /usr/lib/asterisk/modules/chan_skinny.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_skinny.so Reading symbols from /usr/lib/asterisk/modules/chan_oss.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_oss.so Reading symbols from /usr/lib/asterisk/modules/chan_phone.so...done. Loaded symbols for /usr/lib/asterisk/modules/chan_phone.so Reading symbols from /usr/lib/asterisk/modules/pbx_config.so...done. Loaded symbols for /usr/lib/asterisk/modules/pbx_config.so Reading symbols from /usr/lib/asterisk/modules/pbx_wilcalu.so...done. Loaded symbols for /usr/lib/asterisk/modules/pbx_wilcalu.so Reading symbols from /usr/lib/asterisk/modules/pbx_spool.so...done. Loaded symbols for /usr/lib/asterisk/modules/pbx_spool.so Reading symbols from /usr/lib/asterisk/modules/app_dial.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_dial.so Reading symbols from /usr/lib/asterisk/modules/app_playback.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_playback.so Reading symbols from /usr/lib/asterisk/modules/app_voicemail.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_voicemail.so Reading symbols from /usr/lib/asterisk/modules/app_directory.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_directory.so Reading symbols from /usr/lib/asterisk/modules/app_mp3.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_mp3.so Reading symbols from /usr/lib/asterisk/modules/app_system.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_system.so ---Type to continue, or q to quit--- Reading symbols from /usr/lib/asterisk/modules/app_echo.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_echo.so Reading symbols from /usr/lib/asterisk/modules/app_record.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_record.so Reading symbols from /usr/lib/asterisk/modules/app_image.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_image.so Reading symbols from /usr/lib/asterisk/modules/app_url.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_url.so Reading symbols from /usr/lib/asterisk/modules/app_disa.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_disa.so Reading symbols from /usr/lib/asterisk/modules/app_agi.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_agi.so Reading symbols from /usr/lib/asterisk/modules/app_qcall.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_qcall.so Reading symbols from /usr/lib/asterisk/modules/app_adsiprog.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_adsiprog.so Reading symbols from /usr/lib/asterisk/modules/app_getcpeid.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_getcpeid.so Reading symbols from /usr/lib/asterisk/modules/app_milliwatt.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_milliwatt.so Reading symbols from /usr/lib/asterisk/modules/app_zapateller.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_zapateller.so Reading symbols from /usr/lib/asterisk/modules/app_datetime.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_datetime.so Reading symbols from /usr/lib/asterisk/modules/app_setcallerid.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_setcallerid.so Reading symbols from /usr/lib/asterisk/modules/app_festival.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_festival.so Reading symbols from /usr/lib/asterisk/modules/app_queue.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_queue.so Reading symbols from /usr/lib/asterisk/modules/app_senddtmf.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_senddtmf.so Reading symbols from /usr/lib/asterisk/modules/app_parkandannounce.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_parkandannounce.so Reading symbols from /usr/lib/asterisk/modules/app_striplsd.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_striplsd.so Reading symbols from /usr/lib/asterisk/modules/app_setcidname.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_setcidname.so Reading symbols from /usr/lib/asterisk/modules/app_lookupcidname.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_lookupcidname.so Reading symbols from /usr/lib/asterisk/modules/app_substring.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_substring.so Reading symbols from /usr/lib/asterisk/modules/app_macro.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_macro.so Reading symbols from /usr/lib/asterisk/modules/app_authenticate.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_authenticate.so ---Type to continue, or q to quit--- Reading symbols from /usr/lib/asterisk/modules/app_softhangup.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_softhangup.so Reading symbols from /usr/lib/asterisk/modules/app_lookupblacklist.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_lookupblacklist.so Reading symbols from /usr/lib/asterisk/modules/app_waitforring.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_waitforring.so Reading symbols from /usr/lib/asterisk/modules/app_privacy.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_privacy.so Reading symbols from /usr/lib/asterisk/modules/app_db.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_db.so Reading symbols from /usr/lib/asterisk/modules/app_chanisavail.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_chanisavail.so Reading symbols from /usr/lib/asterisk/modules/app_enumlookup.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_enumlookup.so Reading symbols from /usr/lib/asterisk/modules/app_transfer.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_transfer.so Reading symbols from /usr/lib/asterisk/modules/app_setcidnum.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_setcidnum.so Reading symbols from /usr/lib/asterisk/modules/app_cdr.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_cdr.so Reading symbols from /usr/lib/asterisk/modules/app_hasnewvoicemail.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_hasnewvoicemail.so Reading symbols from /usr/lib/asterisk/modules/app_sayunixtime.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_sayunixtime.so Reading symbols from /usr/lib/asterisk/modules/app_cut.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_cut.so Reading symbols from /usr/lib/asterisk/modules/app_read.so...done. Loaded symbols for /usr/lib/asterisk/modules/app_read.so Reading symbols from /usr/lib/asterisk/modules/codec_ilbc.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_ilbc.so Reading symbols from /usr/lib/asterisk/modules/codec_gsm.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_gsm.so Reading symbols from /usr/lib/asterisk/modules/codec_adpcm.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_adpcm.so Reading symbols from /usr/lib/asterisk/modules/codec_ulaw.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_ulaw.so Reading symbols from /usr/lib/asterisk/modules/codec_alaw.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_alaw.so Reading symbols from /usr/lib/asterisk/modules/codec_a_mu.so...done. Loaded symbols for /usr/lib/asterisk/modules/codec_a_mu.so Reading symbols from /usr/lib/asterisk/modules/format_gsm.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_gsm.so Reading symbols from /usr/lib/asterisk/modules/format_wav.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_wav.so Reading symbols from /usr/lib/asterisk/modules/format_wav_gsm.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_wav_gsm.so ---Type to continue, or q to quit--- Reading symbols from /usr/lib/asterisk/modules/format_vox.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_vox.so Reading symbols from /usr/lib/asterisk/modules/format_pcm.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_pcm.so Reading symbols from /usr/lib/asterisk/modules/format_g729.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_g729.so Reading symbols from /usr/lib/asterisk/modules/format_pcm_alaw.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_pcm_alaw.so Reading symbols from /usr/lib/asterisk/modules/format_h263.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_h263.so Reading symbols from /usr/lib/asterisk/modules/format_jpeg.so...done. Loaded symbols for /usr/lib/asterisk/modules/format_jpeg.so Reading symbols from /usr/lib/asterisk/modules/cdr_csv.so...done. Loaded symbols for /usr/lib/asterisk/modules/cdr_csv.so #0 0x400c787e in sigsuspend () from /lib/libc.so.6 (gdb) bt #0 0x400c787e in sigsuspend () from /lib/libc.so.6 #1 0x40022879 in __pthread_wait_for_restart_signal () from /lib/libpthread.so.0 #2 0x40024a36 in __pthread_alt_lock () from /lib/libpthread.so.0 #3 0x40020fbd in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x0809970c in ast_autoservice_stop (chan=0x8879728) at include/asterisk/lock.h:89 #5 0x08082e7c in ast_rtp_bridge (c0=0x40810d20, c1=0x8879728, flags=0, fo=0xfffffffc, rc=0xfffffffc) at rtp.c:1159 #6 0x0805f77c in ast_channel_bridge (c0=0x40810d20, c1=0x8879728, flags=0, fo=0xb6bfee04, rc=0xb6bfee08) at channel.c:2221 #7 0x401d007a in ast_bridge_call (chan=0x40810d20, peer=0x8879728, allowredirect_in=0, allowredirect_out=0, allowdisconnect=0) at res_parking.c:224 #8 0x406da4bd in dial_exec (chan=0x40810d20, data=0x0) at app_dial.c:676 #9 0x08068352 in pbx_exec (c=0x40810d20, app=0x81050a0, data=0xb6bff79c, newstack=1) at pbx.c:396 #10 0x0807239f in pbx_extension_helper (c=0x40810d20, context=0x40810e88 "default", exten=0x40810f7c "5559995785", priority=1, callerid=0x408128e0 "5127351473", action=135286944) at pbx.c:1170 #11 0x0806acaa in ast_pbx_run (c=0x40810d20) at pbx.c:1654 #12 0x080731a1 in pbx_thread (data=0xfffffffc) at pbx.c:1875 #13 0x400200ba in pthread_start_thread () from /lib/libpthread.so.0 #14 0x4016dd6a in clone () from /lib/libc.so.6 (gdb) fr 4 #4 0x0809970c in ast_autoservice_stop (chan=0x8879728) at include/asterisk/lock.h:89 89 res = pthread_mutex_lock(&t->mutex); (gdb) l 84 return res; 85 } 86 87 static inline int __ast_pthread_mutex_lock(char *filename, int lineno, char *func, ast_mutex_t *t) { 88 int res; 89 res = pthread_mutex_lock(&t->mutex); 90 if (!res) { 91 t->file = filename; 92 t->lineno = lineno; 93 t->func = func; (gdb) print (ast_mutex_t)t $1 = {mutex = {__m_reserved = 2051, __m_count = 0, __m_owner = 0x5, __m_kind = 1, __m_lock = {__status = 7, __spinlock = 0}}, file = 0x0, lineno = 0, func = 0x0, thread = 0} (gdb) ####################################################################### END DEUBG SESSION ####################################################################### ################ LOGFILE ################# Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 549 (__sip_ack): Acked pending invite 102 Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 567 (__sip_ack): Stopping retransmission on '973245_13274892@10.31.14.253' of Request 102: Found Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 3417 (build_route): build_route: Contact hop: Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 549 (__sip_ack): Acked pending invite 103 Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 567 (__sip_ack): Stopping retransmission on '0d533ca440a0734c5585c3b847946e35@10.31.4.60' of Request 103: Found Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 3417 (build_route): build_route: Contact hop: Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 1614 (sip_alloc): Allocating new SIP call for 39dd97fd4bf7a27f12b1f94d0061a9f4@10.31.4.60 Jan 9 04:37:14 DEBUG[6151]: File chan_sip.c, Line 5020 (handle_request): That's odd... Got a response on a call we dont know about. Jan 9 04:37:14 DEBUG[33665097]: File rtp.c, Line 1008 (ast_rtp_write): Ooh, format changed from UNKN to ULAW Jan 9 04:37:14 DEBUG[33665097]: File rtp.c, Line 943 (ast_rtp_raw_write): Difference is 4544, ms is 588 **************** END LOGFILE ################### ######################### ASTERISK PROGRAM OUTPUT ######################## Urgent handler Urgent handler Urgent handler -- Called 5559995776@10.31.15.253:4000 Urgent handler -- Called 5559995785@10.31.15.253:4000 Urgent handler -- Called 5559995826@10.31.15.253:4000 Urgent handler -- Called 5559995832@10.31.15.253:4000 -- SIP/10.31.15.253-a777 is ringing Urgent handler -- Called 5559995845@10.31.15.253:4000 -- SIP/10.31.15.253-a777 answered SIP/-40806658 -- Attempting native bridge of SIP/-40806658 and SIP/10.31.15.253-a777 Urgent handler -- Called 5559995851@10.31.15.253:4000 -- SIP/10.31.15.253-2e04 is ringing -- SIP/10.31.15.253-2e04 answered SIP/-4080c7a8 -- Attempting native bridge of SIP/-4080c7a8 and SIP/10.31.15.253-2e04 -- SIP/10.31.15.253-e339 is ringing WARNING[33665097]: File channel.c, Line 1207 (ast_indicate): Unable to handle indication 3 for 'SIP/-40800440' -- SIP/10.31.15.253-e339 answered SIP/-40800440 -- Attempting native bridge of SIP/-40800440 and SIP/10.31.15.253-e339 -- SIP/10.31.15.253-8698 is ringing WARNING[33666122]: File channel.c, Line 1207 (ast_indicate): Unable to handle indication 3 for 'SIP/-40a063d8' Urgent handler -- Called 5559995801@10.31.15.253:4000 -- SIP/10.31.15.253-8698 answered SIP/-40a063d8 -- Attempting native bridge of SIP/-40a063d8 and SIP/10.31.15.253-8698 -- SIP/10.31.15.253-ddfa is ringing -- SIP/10.31.15.253-ddfa answered SIP/-40a0c570 -- Attempting native bridge of SIP/-40a0c570 and SIP/10.31.15.253-ddfa -- SIP/10.31.15.253-4c9d is ringing Urgent handler -- Called 5559995842@10.31.15.253:4000 -- SIP/10.31.15.253-268f is ringing -- SIP/10.31.15.253-4c9d answered SIP/-40aeb0f8 -- Attempting native bridge of SIP/-40aeb0f8 and SIP/10.31.15.253-4c9d -- SIP/10.31.15.253-268f answered SIP/-40af13e0 -- Attempting native bridge of SIP/-40af13e0 and SIP/10.31.15.253-268f -- SIP/10.31.15.253-1679 is ringing -- SIP/10.31.15.253-1679 answered SIP/-408ead28 -- Attempting native bridge of SIP/-408ead28 and SIP/10.31.15.253-1679 Urgent handler Urgent handler WARNING[33663047]: File channel.c, Line 843 (ast_waitfor_nandfds): Thread 33663047 Blocking 'SIP/10.31.15.253-a777', already blocked b 15376 in procedure ast_waitfor_nandfds !! Forcing immediate crash a-la abort !! Urgent handler Urgent handler Urgent handler Segmentation fault (core dumped) ################################# END ASTERISK OUTPUT #################################