[Home]

Summary:ASTERISK-11443: memory leak
Reporter:destiny6628 (destiny6628)Labels:
Date Opened:2008-02-15 01:20:50.000-0600Date Closed:2008-04-17 12:30:50
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 10.0.0.5_console_memory_corruption.txt
( 1) 11999.patch
( 2) 1_PRI_SHOW_MEMORY_SUMMARY_OUTPUT.rar
( 3) 2_PRU_SHOW_MEMORY_OUTPUT.rar
( 4) 3rdtimecoredump.txt
( 5) 4thtimecoredump.txt
( 6) anothercoredumpondifferentserver.txt
( 7) asteriskcrash1pri.txt
( 8) asteriskcrash2pri.txt
( 9) asteriskcrashpri12ndtime.txt
(10) bt_and_bt_full_16_th_.txt
(11) console_output6thmarch.txt
(12) core.txt
(13) core1sttime15thapril.txt
(14) core2ndCHANNELLOCKINGISSUEtime15thapril.txt
(15) core3rdtime15april.txt
(16) core4thtime15thapril.txt
(17) core5thtime15thapril.txt
(18) cored.txt
(19) coredump.txt
(20) coredump1stapril.txt
(21) core_dump_console_output-31st_march.txt
(22) coresvn.txt
(23) frame.txt
(24) framedrop.txt
(25) gaurav.txt
(26) gauravv5th_march.txt
(27) memory_show_summary.txt
(28) module_show.txt
(29) onceinaday.txt
(30) queueframe.txt
(31) show_memory_summary.txt
(32) show_memory_summary1_.txt
(33) show_memory_summary10.0.0.4.txt
(34) show_memory_summaryasterisk.txt
(35) strln.txt
(36) valgrind.rar
(37) wav.txt
(38) wavcore.txt
Description:hi we are using asterisk 1.4.18 , zaptel-1.4.8 and libpri 1.4.3 with 4 e1 card on ibm x3200 server and 90 active calls running on asterisk cli and after some hours of dialing , swap memory starts getting used and asterisk gives core dump .

Asterisk is running with safe_asterisk.

vmstat ouput is as follows

[root@localhost cron]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
1  0    568  57308  44708 1848488    0    0    10   119 1293  509  2  2 95  0  0

Free -m is as follows

[root@localhost cron]# free -m
            total       used       free     shared    buffers     cached
Mem:          2025       1974         51          0         43       1806
-/+ buffers/cache:        124       1901
Swap:         1983          0       1983


OPERATING SYSTEM is centos 5 and kernel version is [root@localhost cron]# uname -a
Linux localhost.localdomain 2.6.18-8.el5 #1 SMP Thu Mar 15 19:57:35 EDT 2007 i686 i686 i386 GNU/Linux

Cannot do valgrind test as it put heavy load on the server and its a production server .

RAM On the server is 2GB .

Only Asterisk application is running on that .




****** ADDITIONAL INFORMATION ******

When asterisk gets crashed core dump is

   -- Executing [9119868435715@default:4] Dial("Local/9119868435715@default-e78                                                                             c,2", "Zap/g1/9868435715||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9868435715
   -- Zap/4-1 is proceeding passing it to Local/9119891310781@default-c5cb,2
   -- Zap/9-1 is proceeding passing it to Local/9119868435715@default-e78c,2
   -- Channel 0/1, span 1 got hangup request, cause 19
   -- Hungup 'Zap/1-1'
 == Everyone is busy/congested at this time (1:0/0/1)
   -- Executing [9119811323149@default:5] Hangup("Local/9119811323149@default-9                                                                             28d,2", "") in new stack
 == Spawn extension (default, 9119811323149, 5) exited non-zero on 'Local/91198                                                                             11323149@default-928d,2'
   -- Executing [h@default:1] DeadAGI("Local/9119811323149@default-928d,2", "ca                                                                             ll_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [h@default:2] DeadAGI("Local/9119811323149@default-928d,2", "ha                                                                             ngup|10.0.0.5|10.0.0.190|19|CHANUNAVAIL|n") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/hangup
   -- AGI Script hangup completed, returning 0
   -- Hungup 'Zap/9-1'
 == Everyone is busy/congested at this time (1:0/0/1)
   -- Executing [9119868435715@default:5] Hangup("Local/9119868435715@default-e                                                                             78c,2", "") in new stack
 == Spawn extension (default, 9119868435715, 5) exited non-zero on 'Local/91198                                                                             68435715@default-e78c,2'
   -- Executing [h@default:1] DeadAGI("Local/9119868435715@default-e78c,2", "ca                                                                             ll_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [h@default:2] DeadAGI("Local/9119868435715@default-e78c,2", "ha                                                                             ngup|10.0.0.5|10.0.0.190|16|CHANUNAVAIL|n") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/hangup
   -- AGI Script hangup completed, returning 0
   -- Executing [9119811370729@default:1] AGI("Local/9119811370729@default-9ec6                                                                             ,2", "call_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [9119811370729@default:2] GotoIf("Local/9119811370729@default-9                                                                             ec6,2", "0?3:4") in new stack
   -- Goto (default,9119811370729,4)
   -- Executing [9119811370729@default:4] Dial("Local/9119811370729@default-9ec                                                                             6,2", "Zap/g1/9811370729||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9811370729
   -- Executing [9119818944882@default:1] AGI("Local/9119818944882@default-5b60                                                                             ,2", "call_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [9119818944882@default:2] GotoIf("Local/9119818944882@default-5                                                                             b60,2", "0?3:4") in new stack
   -- Goto (default,9119818944882,4)
   -- Executing [9119818944882@default:4] Dial("Local/9119818944882@default-5b6                                                                             0,2", "Zap/g1/9818944882||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9818944882
   -- Executing [9119818813050@default:1] AGI("Local/9119818813050@default-055a                                                                             ,2", "call_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- Zap/1-1 is proceeding passing it to Local/9119811370729@default-9ec6,2
   -- AGI Script call_log completed, returning 0
   -- Executing [9119818813050@default:2] GotoIf("Local/9119818813050@default-0                                                                             55a,2", "0?3:4") in new stack
   -- Goto (default,9119818813050,4)
   -- Executing [9119818813050@default:4] Dial("Local/9119818813050@default-055                                                                             a,2", "Zap/g1/9818813050||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9818813050
   -- Zap/9-1 is proceeding passing it to Local/9119818944882@default-5b60,2
   -- Executing [9119818458821@default:1] AGI("Local/9119818458821@default-523c                                                                             ,2", "call_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [9119818458821@default:2] GotoIf("Local/9119818458821@default-5                                                                             23c,2", "0?3:4") in new stack
   -- Goto (default,9119818458821,4)
   -- Executing [9119818458821@default:4] Dial("Local/9119818458821@default-523                                                                             c,2", "Zap/g1/9818458821||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9818458821
   -- Hungup 'Zap/28-1'
 == Spawn extension (default, 09212573648, 4) exited non-zero on 'SIP/222-b6605                                                                             6e8'
   -- Executing [h@default:1] DeadAGI("SIP/222-b66056e8", "call_log|10.0.0.5|10                                                                             .0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [h@default:2] DeadAGI("SIP/222-b66056e8", "hangup|10.0.0.5|10.0                                                                             .0.190|16|ANSWER|n") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/hangup
   -- Executing [9119891116152@default:1] AGI("Local/9119891116152@default-fa2c                                                                             ,2", "call_log|10.0.0.5|10.0.0.190") in new stack
   -- AGI Script hangup completed, returning 0
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [9119891116152@default:2] GotoIf("Local/9119891116152@default-f                                                                             a2c,2", "0?3:4") in new stack
   -- Goto (default,9119891116152,4)
   -- Executing [9119891116152@default:4] Dial("Local/9119891116152@default-fa2                                                                             c,2", "Zap/g1/9891116152||o") in new stack
   -- Requested transfer capability: 0x00 - SPEECH
   -- Called g1/9891116152
   -- Zap/24-1 is proceeding passing it to Local/9119818813050@default-055a,2
   -- Zap/29-1 is proceeding passing it to Local/9119818458821@default-523c,2
   -- Zap/30-1 is proceeding passing it to Local/9119891116152@default-fa2c,2
   -- Zap/24-1 is ringing
   -- Zap/29-1 is ringing
   -- Zap/4-1 is ringing
   -- Zap/4-1 is making progress passing it to Local/9119891310781@default-c5cb                                                                             ,2
   -- Channel 0/30, span 1 got hangup request, cause 31
   -- Hungup 'Zap/30-1'
 == Everyone is busy/congested at this time (1:0/0/1)
   -- Executing [9119891116152@default:5] Hangup("Local/9119891116152@default-f                                                                             a2c,2", "") in new stack
 == Spawn extension (default, 9119891116152, 5) exited non-zero on 'Local/91198                                                                             91116152@default-fa2c,2'
   -- Executing [h@default:1] DeadAGI("Local/9119891116152@default-fa2c,2", "ca                                                                             ll_log|10.0.0.5|10.0.0.190") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/call_log
   -- AGI Script call_log completed, returning 0
   -- Executing [h@default:2] DeadAGI("Local/9119891116152@default-fa2c,2", "ha                                                                             ngup|10.0.0.5|10.0.0.190|31|CHANUNAVAIL|n") in new stack
   -- Launched AGI Script /var/lib/asterisk/agi-bin/hangup
   -- AGI Script hangup completed, returning 0
   -- Zap/6-1 is making progress passing it to Local/9119313568411@default-4ce2                                                                             ,2
   -- Zap/6-1 is busy
   -- Hungup 'Zap/6-1'
 == Everyone is busy/congested at this time (1:1/0/0)
localhost*CLI> *** glibc detected *** /usr/sbin/asterisk: double free or corrupt                                                                             ion (!prev): 0x08a073a0 ***
======= Backtrace: =========
/lib/libc.so.6[0x8edf5d]
/lib/libc.so.6(cfree+0x90)[0x8f15b0]
/usr/sbin/asterisk(ast_frame_free+0x11a)[0x80a2d87]
/usr/sbin/asterisk(__ast_request_and_dial+0x3cc)[0x80855ac]
/usr/sbin/asterisk(ast_pbx_outgoing_exten+0x8e)[0x80c643d]
/usr/sbin/asterisk[0x80b5ac7]
/usr/sbin/asterisk[0x80fd394]
/lib/libpthread.so.0[0x9fb2db]
/lib/libc.so.6(clone+0x5e)[0x95512e]
======= Memory map: ========
00110000-0011a000 r-xp 00000000 fd:00 13241346   /usr/lib/asterisk/modules/res_a                                                                             gi.so
0011a000-00120000 rwxp 0000a000 fd:00 13241346   /usr/lib/asterisk/modules/res_a                                                                             gi.so
00120000-00122000 r-xp 00000000 fd:00 13241480   /usr/lib/asterisk/modules/func_                                                                             groupcount.so
00122000-00123000 rwxp 00001000 fd:00 13241480   /usr/lib/asterisk/modules/func_                                                                             groupcount.so
00123000-00125000 r-xp 00000000 fd:00 13241404   /usr/lib/asterisk/modules/app_m                                                                             illiwatt.so
00125000-00126000 rwxp 00001000 fd:00 13241404   /usr/lib/asterisk/modules/app_m                                                                             illiwatt.so
00126000-00128000 r-xp 00000000 fd:00 13241435   /usr/lib/asterisk/modules/app_v                                                                             erbose.so
00128000-00129000 rwxp 00001000 fd:00 13241435   /usr/lib/asterisk/modules/app_v                                                                             erbose.so
00129000-0012b000 r-xp 00000000 fd:00 13241487   /usr/lib/asterisk/modules/func_                                                                             realtime.so
0012b000-0012c000 rwxp 00001000 fd:00 13241487   /usr/lib/asterisk/modules/func_                                                                             realtime.so
0012c000-0012e000 r-xp 00000000 fd:00 13241420   /usr/lib/asterisk/modules/app_s                                                                             enddtmf.so
0012e000-0012f000 rwxp 00001000 fd:00 13241420   /usr/lib/asterisk/modules/app_s                                                                             enddtmf.so
0012f000-00131000 r-xp 00000000 fd:00 13241382   /usr/lib/asterisk/modules/app_d                                                                             b.so
00131000-00132000 rwxp 00001000 fd:00 13241382   /usr/lib/asterisk/modules/app_d                                                                             b.so
00132000-00134000 r-xp 00000000 fd:00 13241396   /usr/lib/asterisk/modules/app_g                                                                             etcpeid.so
00134000-00135000 rwxp 00001000 fd:00 13241396   /usr/lib/asterisk/modules/app_g                                                                             etcpeid.so
00135000-0013c000 r-xp 00000000 fd:00 13241363   /usr/lib/asterisk/modules/chan_                                                                             phone.so
0013c000-0013d000 rwxp 00007000 fd:00 13241363   /usr/lib/asterisk/modules/chan_                                                                             phone.so
0013d000-0013f000 r-xp 00000000 fd:00 13241477   /usr/lib/asterisk/modules/func_                                                                             enum.so
0013f000-00140000 rwxp 00001000 fd:00 13241477   /usr/lib/asterisk/modules/func_                                                                             enum.so
00140000-00143000 r-xp 00000000 fd:00 13241461   /usr/lib/asterisk/modules/forma                                                                             t_ogg_vorbis.so
00143000-00144000 rwxp 00002000 fd:00 13241461   /usr/lib/asterisk/modules/forma                                                                             t_ogg_vorbis.so
00144000-00146000 r-xp 00000000 fd:00 13241412   /usr/lib/asterisk/modules/app_p                                                                             rivacy.so
00146000-00147000 rwxp 00001000 fd:00 13241412   /usr/lib/asterisk/modules/app_p                                                                             rivacy.so
00148000-0014f000 r-xp 00000000 fd:00 13241353   /usr/lib/asterisk/modules/res_m                                                                             usiconhold.so
0014f000-00150000 rwxp 00007000 fd:00 13241353   /usr/lib/asterisk/modules/res_m                                                                             usiconhold.so
00150000-00152000 r-xp 00000000 fd:00 13241417   /usr/lib/asterisk/modules/app_r                                                                             ealtime.so
00152000-00153000 rwxp 00002000 fd:00 13241417   /usr/lib/asterisk/modules/app_r                                                                             ealtime.so
00153000-00155000 r-xp 00000000 fd:00 13241490   /usr/lib/asterisk/modules/func_                                                                             timeout.so
00155000-00156000 rwxp 00001000 fd:00 13241490   /usr/lib/asterisk/modules/func_                                                                             timeout.so
00156000-00157000 r-xp 00000000 fd:00 13241451   /usr/lib/asterisk/modules/codec                                                                             _ulaw.so
00157000-00158000 rwxp 00001000 fd:00 13241451   /usr/lib/asterisk/modules/codec                                                                             _ulaw.so
00158000-00173000 r-xp 00000000 fd:00 13241369   /usr/lib/asterisk/modules/pbx_d                                                                             undi.so
00173000-00175000 rwxp 0001a000 fd:00 13241369   /usr/lib/asterisk/modules/pbx_d                                                                             undi.so
00175000-0017e000 r-xp 00000000 fd:00 13241373   /usr/lib/asterisk/modules/app_a                                                                             dsiprog.so
0017e000-0017f000 rwxp 00009000 fd:00 13241373   /usr/lib/asterisk/modules/app_a                                                                             dsiprog.so
0017f000-00181000 r-xp 00000000 fd:00 13241426   /usr/lib/asterisk/modules/app_s                                                                             ofthangup.so
00181000-00182000 rwxp 00001000 fd:00 13241426   /usr/lib/asterisk/modules/app_s                                                                             of
Disconnected from Asterisk server
Executing last minute cleanups
Asterisk cleanly ending (0).
[root@localhost 20080215]#
Comments:By: Mark Michelson (mmichelson) 2008-02-15 14:58:34.000-0600

If you can't use valgrind on the system, then perhaps you can compile with the MALLOC_DEBUG compiler flag enabled in menuselect. This keeps track of memory allocations and provides two new commands, "memory show allocations" and "memory show summary". By running these commands somewhat regularly, it may be possible to analyze the output and zero in on where the leak is occurring.
By: destiny6628 (destiny6628) 2008-02-16 03:08:53.000-0600

Sure will try same on the server and will provide the feedback on same .

However today also after 4 hours of calling Asterisk gave core dump and got crashed twice in a span of 45 min each and everytime core dump was more or less was same on "Wait for Answer" and status as NO ANSWER

I am attaching the bt and bt full as well.
By: destiny6628 (destiny6628) 2008-02-19 03:56:15.000-0600

Hi today i have compiled with the malloc_debug and also tried even running valgrind test but that took the load average very high which caused servere voice breakage .

So i stopped valgrind test but have got the 2 new commands and attaching the output as well .
By: jmls (jmls) 2008-02-19 15:46:24.000-0600

destiny6628, is your box still running ? If so, could you upload a new show memory summary file ? Thanks
By: destiny6628 (destiny6628) 2008-02-20 00:15:21.000-0600

Hi i am attaching the output of show memory summary as well .

vmstat is as follows :---

[root@localhost tmp]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
2  0    584 1464752  22028 444412    0    0     7   129  235   97  2 22 76  1  0

free -m is as follows :---

[root@localhost tmp]# free -m
            total       used       free     shared    buffers     cached
Mem:          2025        611       1414          0         21        449
-/+ buffers/cache:        140       1885
Swap:         1983          0       1983
By: destiny6628 (destiny6628) 2008-02-20 00:38:15.000-0600

There is another server as well where we 2 Pri's are terminated on 4 e1 card and got the core dump as well .


top output is as follows :---

top - 12:09:32 up  3:24,  3 users,  load average: 0.99, 0.71, 0.58
Tasks:  82 total,   1 running,  81 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.2%us,  2.4%sy,  0.0%ni, 94.5%id,  0.6%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   2074484k total,  1324180k used,   750304k free,    87108k buffers
Swap:  2031608k total,        0k used,  2031608k free,  1151436k cached

----------------------------------------------

free -m as follows :---

[root@localhost tmp]# free -m
            total       used       free     shared    buffers     cached
Mem:          2025       1304        721          0         85       1135
-/+ buffers/cache:         83       1942
Swap:         1983          0       1983

--------------------------------------------------

[root@localhost tmp]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0      0 733448  87320 1167832    0    0    11   149 1374  458  2  3 94  1  0

--------------------------------------------------------


Show memory summary is also attached by the name of show memory summary 1.txt
By: destiny6628 (destiny6628) 2008-02-20 00:40:52.000-0600

We got another server as well where only 1 Pri is terminated and there also we got the asterisk core dump .

Top output is as follows

top - 12:13:08 up 23:04,  3 users,  load average: 0.38, 0.47, 0.45
Tasks:  62 total,   1 running,  61 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.5% us,  1.9% sy,  0.0% ni, 62.5% id,  0.4% wa, 32.5% hi,  0.3% si
Mem:   2059136k total,  1999664k used,    59472k free,    20208k buffers
Swap:   491512k total,      260k used,   491252k free,  1846156k cached

-----------------------------------------------

free -m is as follows

[root@localhost tmp]# free -m
            total       used       free     shared    buffers     cached
Mem:          2010       1956         54          0         19       1806
-/+ buffers/cache:        129       1881
Swap:          479          0        479

-------------------------------------------------


vmstat is as follows :----

[root@localhost tmp]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
1  0    260  52108  20328 1853512    0    0     4   144  283   268  2 35 62  0


------------------------------------------------------------


show memory summary is also attached by the name of show memory summary 10.0.0.4.txt .
By: destiny6628 (destiny6628) 2008-02-20 01:34:32.000-0600

Again on the Server where 1 Pri is terminated Asterisk got killed again .

Attaching core dump by the name of asteriskcrashpri12ndtime.

Show memory summary by the name of showmemorysummaryasterisk.txt

vmstat is as follows :---

[root@localhost tmp]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
0  0    248  97500  24520 1793056    0    0     4   151  379   387  3 35 62  0

------------------------------------


free -m

[root@localhost tmp]# free -m
            total       used       free     shared    buffers     cached
Mem:          2010       1915         94          0         23       1751
-/+ buffers/cache:        140       1870
Swap:          479          0        479


------------------------------------

top is as follows :---

top - 13:07:46 up 23:59,  3 users,  load average: 0.02, 0.32, 0.50
Tasks:  63 total,   1 running,  62 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.6% us,  2.0% sy,  0.0% ni, 61.9% id,  0.4% wa, 32.8% hi,  0.3% si
Mem:   2059136k total,  1963108k used,    96028k free,    24580k buffers
Swap:   491512k total,      248k used,   491264k free,  1794068k cached

------------------------------------------------
By: Mark Michelson (mmichelson) 2008-02-20 10:28:10.000-0600

While I appreciate the amount of data you're providing, it's not especially helpful. When diagnosing a memory leak, what we need is the output of several "memory show allocations" and "memory show summary" on the same running process. The idea is that we can see a change in the number of allocations over time and possibly pinpoint where the memory leak(s) is. Outputting a single memory summary for a process is not helpful since it gives no reference point for the memory allocations later in the process.

Also, given the crashes experienced, I have a feeling that this is more than just a memory leak and that there's some sort of corruption happening here. There are a few similar issues on the tracker right now, and it would be nice if we could figure out what the common elements are in the setups.
By: David Brillert (aragon) 2008-02-20 10:45:24.000-0600

1.4.18
Centos 4.6

I can reproduce Asterisk crash by doing pages to SIP phones on two different servers.
While running top I can see free memory dropping with each successive page until safe_asterisk restarts my system.
I dont have valgrind installed or dont optimize enabled etc... so I cant be much help with backtraces but the crash is definitely reproduced with paging.
I only have 4 phones in my page group so I page successively about 60 times to reproduce the crash.
By: destiny6628 (destiny6628) 2008-02-20 10:48:05.000-0600

Thanks putnopvut for the instant reply back , i got your point completely .Will have the production server online tomorrow , will get as many show memory output and lets see if that can help us .

Thanks once again .
By: destiny6628 (destiny6628) 2008-02-21 06:09:59.000-0600

hi i have taken show memory summary in interval of hour and half n hour from 2 servers .

On one PRI is terminated

On other 2 PRI's are terminated .


The server which has 2 PRI's gave core dump today which seems it got some problem while writing a string length
.

Attached is the core dump as well by the name of

strln.txt.


Attaching zip file by name of  show memory summary output of 1 PRI server  and also of 2 PRI server .

Lets hope this help us .
By: destiny6628 (destiny6628) 2008-02-25 10:06:15.000-0600

I am still waiting for any help if any one can offer me ??

Thanks in advance for same .
By: Russell Bryant (russell) 2008-02-25 17:59:13.000-0600

I don't see anything in the memory summary that you uploaded that indicates any kind of memory leak.  Everything looks pretty stable in the Asterisk allocations that have been recorded.
By: Russell Bryant (russell) 2008-02-25 18:02:13.000-0600

Can you provide the output of "module show" so that I can see the list of modules that you are using?
By: destiny6628 (destiny6628) 2008-02-26 04:12:47.000-0600

Hi

Thanks for the prompt response on the issue .

Am attaching module show output , we are using only SIP , ZAP and ulaw codec in our configuration , nothing else .

Recordings of all the calls are done in wav format .


By: destiny6628 (destiny6628) 2008-02-26 10:30:45.000-0600

I have attached show module , waiting for the reply , if possible please reply on same as we are having some serious problem with asterisk giving core dump on production server .
By: jmls (jmls) 2008-02-26 10:38:33.000-0600

are you sure that it is asterisk causing the swap - from what I've seen, your memory usage for asterisk is around th 20-25MB level. That would not cause asterisk to swap.

Could you post any "top" information showing the processes consuming the memory.
Thanks.
By: Jason Parker (jparker) 2008-02-27 14:30:42.000-0600

Of all of the free and top outputs you've given, swap has never gone above 260k used.  You don't appear to be swapping at all, let alone heavily.  Where are you seeing swap being used?
By: destiny6628 (destiny6628) 2008-03-03 00:48:37.000-0600

Hi ,

Memory leak is fine .There is no problem related to memory leak as of now but still my asterisk is getting crash .

I have also updated the patch of latest auto service.c which was updated Russel .

I am attaching the core dump as well.

core dump is by the name of coredump.txt


By: Mark Michelson (mmichelson) 2008-03-03 16:03:25.000-0600

After reading over the core dumps experienced, I believe that this problem may have already been fixed.

If you could, please try running the latest checkout of Asterisk 1.4. This has several fixes in chan_local (which appears to be the source of the crash) including fixes in the local_queue_frame function to ensure that the channel being referenced has not been freed before attempting to operate on it.
By: destiny6628 (destiny6628) 2008-03-04 12:34:35.000-0600

Have installed asterisk , zaptel and libpri through svn latest check out and hope that issue related to core dump gets resolved .

i ll keep you all updated on same .

Thanks for your always prompt response .
By: destiny6628 (destiny6628) 2008-03-05 01:19:25.000-0600

After doing the svn update as well asterisk gave core dump .

This was because of the ast_channel_free .

I am attaching the core dump as well by the name of gaurav.txt
By: destiny6628 (destiny6628) 2008-03-05 02:21:45.000-0600

Today we had second core dump of the day on the same server after 2 hours of time and reason for the core dump is exactly same like ast_channel_free .

Am attaching the core dump by name of gauravv5thmarch.txt
By: destiny6628 (destiny6628) 2008-03-05 02:22:02.000-0600

#0  0x08082533 in ast_channel_free (chan=0x977fdb0) at channel.c:1234
#1  0x0808305e in ast_hangup (chan=0x977fdb0) at channel.c:1496
#2  0x080c4b21 in __ast_pbx_run (c=0x977fdb0) at pbx.c:2563
#3  0x080c4d09 in pbx_thread (data=0x977fdb0) at pbx.c:2623
#4  0x081014c2 in dummy_start (data=0x97d2ec0) at utils.c:865
ASTERISK-1  0x007f545b in start_thread () from /lib/libpthread.so.0
ASTERISK-2  0x0095624e in clone () from /lib/libc.so.6
By: destiny6628 (destiny6628) 2008-03-05 03:18:00.000-0600

SVN VERSION IS

Connected to Asterisk SVN-branch-1.4-r105591 currently running on localhost (pid                                                                              = 8233)
Verbosity is at least 3
By: destiny6628 (destiny6628) 2008-03-05 05:48:23.000-0600

waiting for reply as we are in a state of crisis due to asterisk getting crash on regular basis due to some reasons .

Have tried the patch and now running on latest svn version as well .

Please suggests what needs to be done more so that thinsg can improve .

This is very critical of us please help .
By: Mark Michelson (mmichelson) 2008-03-05 08:41:19.000-0600

Okay, so it seems like the crash that had been occurring previously (the one in ast_queue_frame called from local_queue_frame) has not happened again, but there's one that's happening now on hangup, and it also has to do with the channel's readq apparently becoming corrupt.

Again, this is happening with a Local channel, so I would guess that there's still some issue in that code. I will look into it further.
By: Mark Michelson (mmichelson) 2008-03-05 08:48:27.000-0600

Oh, and another thing, if you're looking for some sort of temporary workaround, then if at all possible, you can try using some other type of channel instead of Local channels. The crashes you experience are consistently happening with Local channels.
By: Mark Michelson (mmichelson) 2008-03-05 10:44:48.000-0600

I spent a few hours this morning attempting to reproduce this problem by scripting some calls which used Local channels. Unfortunately, no matter what I tried I could not get a crash to occur, and running with valgrind did not show any memory corruption problems.

So it looks like there's some specific set of conditions occurring on your system that is causing this to happen. I'm going to need some information from you. First of all, if possible, can you describe how it is that you're using Local channels on your system? The best way to do this would be to upload the part(s) of your dialplan that use Local channels. Also, one thing that I have not yet seen is any sort of console output prior to a crash. If you could turn up the debug level and verbosity level and upload the lines that print prior to a crash, that may be useful too. The most helpful thing you could do, if at all possible, would be to run valgrind on the system since it seems clear that this is an issue of memory corruption.

Thanks for your patience on this one.
By: destiny6628 (destiny6628) 2008-03-05 23:57:16.000-0600

hi thanks for your prompt reply and appreciate you have taken a time out and looked into our issue .

I ll explain you why we are using local channels and our dial plan as well .

We are using a dialer application where we dial out the local channels and when sip channels gets formed we transfer the call to the executives.

Local channel we are using because we send monitor and then change monitor is required for recording the calls .

exten => _91.,1,AGI(call_log,10.0.0.5|10.0.0.190)
exten => _91.,2,GotoIf($["${CALLFILENAME}" != ""]?3:4)
exten => _91.,3,monitor(wav,/var/spool/asterisk/monitor/ORIG/${STRFTIME(${EPOCH},Asia/Calcutta,%Y%m%d)}/${CALLFILENAME})
exten => _91.,4,Dial(Zap/g1/${EXTEN:3},,o|)
exten => _91.,5,Hangup

Caller id is also required for transferring the call to the agents thats why we are using ,,o as well .

At the starting local channel gets formed and after that SIP or ZAP channel gets takes place and thats why we are using local channel .

We are using httpd and php for the web interface and other pages for dialing and all .

------------------------------------

Today also in the morning we had one core dump but thats was not due to hangup on local channel but due to

Core was generated by `/usr/sbin/asterisk -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
#0  0x08099217 in do_devstate_changes (data=0x0) at devicestate.c:365
365 cur = AST_LIST_REMOVE_HEAD(&state_changes, list);
(gdb) bt
#0  0x08099217 in do_devstate_changes (data=0x0) at devicestate.c:365
#1  0x081014c2 in dummy_start (data=0x886adb8) at utils.c:865
#2  0x007f545b in start_thread () from /lib/libpthread.so.0
#3  0x0095624e in clone () from /lib/libc.so.6

Attaching the core dump as well by the name of core.txt

Will also upload the console output as well if again asterisk gets crashed .

Thanks again for your response .
By: destiny6628 (destiny6628) 2008-03-06 02:32:03.000-0600

Another core dump came on the svn version i have taken the console output as well and uploading the console output by the name of consoleoutput6thmarch.txt and coredump by the name of cored.txt .

May be this can provide some help in solving the problem .

Thanks once again .

Will wait for your response .
By: destiny6628 (destiny6628) 2008-03-06 04:29:42.000-0600

Today 4 times asterisk has got killed .

Twice i have uploaded the core dump and once console output as well.

Twice more core dump has came .

The intensity of the core dump accoridng to me has increased even after upgrading to SVN version .

Much i am not aware off .

I am attaching the core dump by the name of 3rd time and 4time .

Please suggests what needs to be done .
By: jmls (jmls) 2008-03-06 04:48:54.000-0600

this may be so off base I don't want to ask, but, are you using odbc for realtime or func_odbc at all ? If so, which database ?

Do you have another machine that you could put into production just in case there is a hardware problem ? If you have the same problems on a different machine then we could probably say that it is asterisk. If you have no problems, then we could safely say that it was your hardware.

For what it's worth, we are running SVN-branch-1.4-r98849, using a dialler to put calls into a local channel, call a zap channel and then another local channel to place the call with an agent. We make somewhere in the region of 6-7k calls per day, and are currently on an uptime of 5 weeks. The last downtime was planned maintenance.
By: destiny6628 (destiny6628) 2008-03-06 05:26:40.000-0600

I have 3 productions servers one is curently upgraded on SVN version and other 2 are on asterisk-1.4.18 with latest patches .

On the svn version asterisk is getting killed today 4 times it happened .

On the rest also asterisk is getting killed but frequency is once or twice in a day  and core dump is also more or less same .

Let me attach their core dump as well .


By: destiny6628 (destiny6628) 2008-03-06 05:38:52.000-0600

anothercoredumpondifferentserver.txt is the core dump on the server which is of same configuration as the server which is giving 4 times core dump with SVN version .

onceinadaycoredump.txt is a core dump of the another machine which is P4 .

The amount of core dump coming on any server is not equal , sometimes x amount of core dump cames on one server and x amount of core dump comes on another server .

But the reason for coming the core dump is more or less same on each server which is quite evident by the core i have attached .
By: destiny6628 (destiny6628) 2008-03-07 00:52:13.000-0600

I am still waiting for same .Please reply this is urgent for us .
By: jmls (jmls) 2008-03-07 08:59:15.000-0600

hmm, there seems to be three differences to the way that we are using asterisk (local channels / dialler etc)

1) AGI ==> What are you using this for ? What tech is in the AGI ?
2) ,,o (callerid) ==> can you test without this flag ?
3) Monitor with wav ==> can you test MixMonitor with gsm ?
By: Mark Michelson (mmichelson) 2008-03-07 10:13:05.000-0600

Update: I have not found the cause of the crashes you are experiencing. What makes things difficult is that it appears that your crashes are happening all over the place, and it makes it even more difficult to pinpoint the crash cause. Here are some of the observations I've made:

1. The most common causes of crashes seem to come from frame list corruption, either on the channel's readq or another frame list, although some of the crashes you have presented are happening in dsp code or other channel-related code, indicating that perhaps the channel itself is getting corrupted.

2. The cored.txt upload shows for once that there is definitely corruption in the frame list. If you look at the bt full from that upload, the value of cur in frame 0 is 0xdeadbeef, which is a magic value used to separate allocations and should absolutely NEVER show up as an actual value for any variable.

The console output that you uploaded from March 6 is odd because it doesn't show a crash. It actually shows Asterisk exiting cleanly.

The one thing you could do that would be most helpful in resolving this situation is to run just one of your servers under valgrind. We're working very hard trying to find an answer, but when my local tests don't crash, valgrind output from the machine where crashes are occurring is the most helpful thing we can have.

Also, to echo jmls, are you using ODBC at all, and what are you doing in your AGI script(s)?
By: Abhay Gupta (agupta) 2008-03-08 23:04:13.000-0600

Can AGI cause corruption of Asterisk variables ?
By: Abhay Gupta (agupta) 2008-03-08 23:51:07.000-0600

I had a word with destiny6628 . There is no odbc being used .

AGI written in C , is being used for making database connection .

parameter o is being used to allow for callerid being set from application and transmitted accordingly . All selection and processing in application is based on callerid and so it is a important parameter.
By: jmls (jmls) 2008-03-09 09:43:35

I'm not saying that you need to remove the o parameter forever, we are just trying to find this crash. Eliminatng issues helps  - if, for example, you remove the o option and there are no more crashes, it helps the developers narrow the scope of the problem.

This is also the case for AGI - if AGI is removed and there are no more crashes ...
By: destiny6628 (destiny6628) 2008-03-09 22:00:04

i am working out for the other alternative without AGI for some time .

Give me a day time as soon as i work out on that ,  will keep you all posted and updated .

Thanks for being involved for such a long time now .
By: Mark Michelson (mmichelson) 2008-03-10 15:12:21

Today, a bug was fixed which involved channel-related memory corruption. This seemed to show itself under heavy load and could be the cause of your crashes. If you could, please check out SVN revision 107158 and see if that fixes the problems you have been seeing. Thanks!
By: Mark Michelson (mmichelson) 2008-03-10 15:15:41

Actually, you might as well go ahead and update to SVN revision 107161, since another bug was found regarding potentially accessing freed channels.
By: destiny6628 (destiny6628) 2008-03-11 00:49:59

thanks so much for the response and prompt reply back .I will surely go ahead and do same and will update you soon .

Thanks a lot once again for your precious time .
By: destiny6628 (destiny6628) 2008-03-11 09:11:07

Have updsted to latest svn version and will update you all about the status of asterisk crash tomorrow once we start calling on that .

Thanks for being with me for so long .

Its a pleasure to be such a product where we all work together as a team to make it a success .
By: destiny6628 (destiny6628) 2008-03-12 07:09:54

After updating to latest svn update , didnt had the asterisk crash problem through out the day .

Had the uptime of 10 hours on the asterisk .

This is really wonderful change as was struggling for that since long time .

But still i want to observe for one more day before comment on same .

Thanks for all your patience , will keep you all posted .

localhost*CLI> show uptime
System uptime: 10 hours, 19 minutes, 29 seconds
Last reload: 4 hours, 54 minutes, 21 seconds
By: Mark Michelson (mmichelson) 2008-03-12 09:36:30

Wonderful! I'm glad to hear that the crashes stopped happening. I agree that we should keep this open for now in case something else happens though.

Good luck!
By: jmls (jmls) 2008-03-13 13:46:41

destiny6628: do you have any more feedback for us ? Thanks.
By: destiny6628 (destiny6628) 2008-03-14 02:46:06

As far as yesterday is concerned asterisk uptime was 10hrs on svn server and asterisk didnt killed twice in a day .

But today asterisk was killed once again and m attaching the core dump .

The core dump is attached by the name of coresvn.txt .


By: destiny6628 (destiny6628) 2008-03-14 07:14:33

Today at the end of the dialing asterisk got crashed only once and that was the only one core dump which came on the asterisk.

That core dump came after 6 hours of dialing .

After that asterisk uptime was

localhost*CLI> show uptime
System uptime: 4 hours, 56 minutes, 5 seconds
By: Mark Michelson (mmichelson) 2008-03-14 10:29:06

Thanks for the latest backtrace. This new one is very similar to the one in cored.txt.

I think I may know why this crash is happening. I suspect that a thread is being started and quickly stopped before its frame cache can be initialized. I will see if I can find a fix for it.
By: Mark Michelson (mmichelson) 2008-03-14 10:53:07

I discussed my idea with Russell and he showed me that my idea was wrong. I'll continue looking to see if I can find what's wrong here.
By: destiny6628 (destiny6628) 2008-03-15 07:16:10

Today after complete dialing we didnt had a single asterisk crash problem on the server .

Asterisk uptime was well above 10 hours .

Asterisk SVN-branch-1.4-r107290M, Copyright (C) 1999 - 2008 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
 == Parsing '/etc/asterisk/asterisk.conf': Found
 == Parsing '/etc/asterisk/extconfig.conf': Found
Connected to Asterisk SVN-branch-1.4-r107290M currently running on localhost (pid = 2828)
Verbosity is at least 4
localhost*CLI> show uptime
System uptime: 10 hours, 25 minutes, 33 seconds
localhost*CLI> exit

Will observe few more days and will keep you all posted and updated .
By: destiny6628 (destiny6628) 2008-03-18 06:09:52

Today had once asterisk giving core dump on svn version server after 2 days .

The core dump which i have seen today is a complete new core dump which had never came across since now .

I am attaching core dump by name of wavecore.txt .
By: destiny6628 (destiny6628) 2008-03-18 06:10:20

SVN-branch-1.4-r107290M this is SVN version .
By: destiny6628 (destiny6628) 2008-03-20 02:07:08

Hi

Your feedback is long awaited for me .

Please respond

Thanks
By: Mark Michelson (mmichelson) 2008-03-20 10:17:07

I apologize for the late reply, but unfortunately there haven't been any new developments with regards to your crashes. Sorry about that.
By: destiny6628 (destiny6628) 2008-03-20 10:37:34

No problems .Thanks for the reply .Lot of stablity has came with last svn update .

I am pretty optimistic it will get even more stable over a period of time .
By: destiny6628 (destiny6628) 2008-03-24 07:36:36

Hi today after a couple of days core dump came on asterisk svn version .

Core dump is related to the frame drop and i am attaching the core dump by the name of framedrop.txt .

In case its solved or solvable please update me on same .

Thanks for being so patient with me for such a long time .
By: destiny6628 (destiny6628) 2008-03-27 07:32:47

Today also one core dump came on svn version which i am running .

The core dump is attached by the name of queueframe.txt

Thanks !!!!
By: Private Name (falves11) 2008-03-27 07:49:07

I am having the same exact issue, and also I cannot do valgrind or I will be of business.
By: Mark Michelson (mmichelson) 2008-03-27 15:07:47

falves11 are you using SIP channels? If you are, I may have a compromise since you cannot run valgrind. If you can run tcpdump on the server with Asterisk and capture all the SIP traffic and upload the resulting pcap file, I can use a tool to generate a sipp scenario and possibly recreate the crash locally. If I can do that, then I may be able to finally isolate the problem areas and come up with a fix for this longstanding issue.

Of course, if you're not using SIP channels, then I'm at a loss again. This issue is just not one that is solvable through static analysis of the code.
By: Private Name (falves11) 2008-03-27 15:24:53

I am using SIP. I would like to hire an engineer to do this trouble shooting, for a fixed fee, since I am a business man and would not know a "tcdump" if I met it head on. Please if you know a volunteer have him contact me at falves1@hotmail.com. He may take control of the machine until we can reproduce the issue. I can force all my traffic into one box until it crashes.
By: Abhay Gupta (agupta) 2008-03-27 22:15:34

We volunteer to do it for free . I am sending you a mail so that we are able to get this bug resolved .
By: destiny6628 (destiny6628) 2008-03-31 04:41:49

Core dump are still coming on SVN version of asterisk and reason for core dump coming everytime is same as usual which i have uploaded earlier .

Is there any solution for the core dump ??

Thanks
By: destiny6628 (destiny6628) 2008-03-31 04:42:12

core dump is attached by the name of frame.txt
By: destiny6628 (destiny6628) 2008-03-31 05:15:57

Another core dump came on the asterisk svn version which happened due to format_wav.c

attaching the core dump by the name of wav.txt and also consoleoutput as well .

If possible please reply on same.

Thanks


By: destiny6628 (destiny6628) 2008-04-01 04:58:40

Eagerly waiting to hear on same and solution of the core dump .

Thanks
By: destiny6628 (destiny6628) 2008-04-01 05:31:28

Today one again on SVN version asterisk core dump came .

I am attaching the core dump by the name of coredump1stapril.txt .
By: Mark Michelson (mmichelson) 2008-04-01 08:41:18

I was unable to come to work yesterday and so I just now got to read the last five notes on this issue.

The problem is that there is no simple solution for these core dumps you are experiencing. I want very badly to fix this but unfortunately, As I said a few notes ago, this issue is not solvable through static analysis of the code. Since I have not been able to reproduce the issue myself, it means that someone, anyone, who also experiences these same types of core dumps should run valgrind in order to detect where invalid memory accesses are so that we can fix this once and for all.
By: Private Name (falves11) 2008-04-01 08:50:10

I volunteer to provide a hosted box with Centos 5.0 64 Bits so the issue can be reprodiced in Valgrind. I have been running on valgrind with 250 calls and it works. My box has 8 cores and several GB of memory. The Internet access is also plenty.
By: destiny6628 (destiny6628) 2008-04-15 07:30:39

sorry for late reply , was trying for valgrind for so many days and have tried running valgrind today .

Have upgraded asterisk to asterisk-1.4.19 version and also traffic on asterisk has increased , due to which asterisk has given core dump many times 5 times in a day .

The problems are like

channels locking is taking place , show channels becomes unresponsive and calls stops going .

On asterisk it shows congestion , unable to create zap channel type .

Attaching the core dump as well and valgrind.txt as well .

Waiting for reply as m in a very critical situation right now .
By: destiny6628 (destiny6628) 2008-04-15 07:35:14

ASTERISK VERSION IS NOW 1.4.19
By: destiny6628 (destiny6628) 2008-04-15 07:53:07

sorry forgot to attach valgrind.txt
By: Mark Michelson (mmichelson) 2008-04-15 11:44:04

destiny6628: You are awesome! Thank you! The valgrind output is so great because it shows exactly where the problem is. I'll have this fixed as soon as possible.
By: Private Name (falves11) 2008-04-15 12:05:39

Dear Developer, could you apply this also in trunk? sorry of the question makes little sense.
By: Mark Michelson (mmichelson) 2008-04-15 16:01:17

Here's an explanation of what is happening.

When you place a call to a Zap channel, app_dial tries to read a frame from the channel. In the case that the Zap channel was busy, its DSP detects the busy signal and stores a "busy" frame. Chan_zap passes a reference to this frame to app_dial. app_dial's reaction upon seeing a "busy" frame is to hang up the channel. When the channel is hung up, its DSP is freed, meaning that the reference that app_dial has to the frame is no longer valid, so any operations that are made on that frame require reading or writing invalid memory. The fix necessary is a bit bizarre, but because there is precedent for such a change, it shouldn't take too long to make the change.

Thanks for your patience on this and thanks very much for providing the valgrind output. I understand that could put extra load on your already busy systems, but the data that it provides is hands-down more helpful than anything else when it comes to memory corruption errors.
By: Private Name (falves11) 2008-04-15 16:24:29

I think that my bug 12540 is related, although I am only SIP to SIP and H323 to SIP or SIP to H323. Please kindly take a look.
By: Mark Michelson (mmichelson) 2008-04-15 19:02:00

I have uploaded 11999.patch which should fix the problem you have been encountering. See if this improves the situation.

falves11: Since this problem was reported against 1.4, the patch provided is for 1.4. When it gets committed to 1.4, it also will be merged into the trunk branch.
By: destiny6628 (destiny6628) 2008-04-16 07:46:26

Thank you so much for the patch .Asterisk didnt killed during the vigerous dialing today and i think much awaited problem has been solved .

I will closely observe more for couple of days and will keep you all posted on same .

Thanks for always looking at the problem providing the solution at the earliest .

It really feels very great when we have such a great developers looking at the problem and providing the solution .

Thanks once again , i am very glad to see the much improved dialing today .
By: Victor Yure (victoryure) 2008-04-16 13:00:48

We have the same IBM server and have been experiencing the same problem. This patch (11999.patch) fixed it.

Thanks.
By: Private Name (falves11) 2008-04-16 13:07:00

When is it going to be merged into the SVN Trunk?? I don't want to apply the patch but compile the corrected sources.
By: Digium Subversion (svnbot) 2008-04-17 11:23:04

Repository: asterisk
Revision: 114207

U   branches/1.4/include/asterisk/frame.h
U   branches/1.4/main/dsp.c
U   branches/1.4/main/frame.c

------------------------------------------------------------------------
r114207 | mmichelson | 2008-04-17 11:23:03 -0500 (Thu, 17 Apr 2008) | 12 lines

It was possible for a reference to a frame which was part of a freed DSP to still be
referenced, leading to memory corruption and eventual crashes. This code change ensures
that the dsp is freed when we are finished with the frame. This change is very similar
to a change Russell made with translators back a month or so ago.

(closes issue ASTERISK-11443)
Reported by: destiny6628
Patches:
     11999.patch uploaded by putnopvut (license 60)
Tested by: destiny6628, victoryure


------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=114207
By: Digium Subversion (svnbot) 2008-04-17 11:34:59

Repository: asterisk
Revision: 114208

_U  trunk/
U   trunk/include/asterisk/dsp.h
U   trunk/include/asterisk/frame.h
U   trunk/main/dsp.c
U   trunk/main/frame.c

------------------------------------------------------------------------
r114208 | mmichelson | 2008-04-17 11:34:58 -0500 (Thu, 17 Apr 2008) | 20 lines

Merged revisions 114207 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114207 | mmichelson | 2008-04-17 11:28:03 -0500 (Thu, 17 Apr 2008) | 12 lines

It was possible for a reference to a frame which was part of a freed DSP to still be
referenced, leading to memory corruption and eventual crashes. This code change ensures
that the dsp is freed when we are finished with the frame. This change is very similar
to a change Russell made with translators back a month or so ago.

(closes issue ASTERISK-11443)
Reported by: destiny6628
Patches:
     11999.patch uploaded by putnopvut (license 60)
Tested by: destiny6628, victoryure


........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=114208
By: Digium Subversion (svnbot) 2008-04-17 11:43:40

Repository: asterisk
Revision: 114210

_U  branches/1.6.0/
U   branches/1.6.0/include/asterisk/dsp.h
U   branches/1.6.0/include/asterisk/frame.h
U   branches/1.6.0/main/dsp.c
U   branches/1.6.0/main/frame.c

------------------------------------------------------------------------
r114210 | mmichelson | 2008-04-17 11:43:40 -0500 (Thu, 17 Apr 2008) | 28 lines

Merged revisions 114208 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r114208 | mmichelson | 2008-04-17 11:40:12 -0500 (Thu, 17 Apr 2008) | 20 lines

Merged revisions 114207 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114207 | mmichelson | 2008-04-17 11:28:03 -0500 (Thu, 17 Apr 2008) | 12 lines

It was possible for a reference to a frame which was part of a freed DSP to still be
referenced, leading to memory corruption and eventual crashes. This code change ensures
that the dsp is freed when we are finished with the frame. This change is very similar
to a change Russell made with translators back a month or so ago.

(closes issue ASTERISK-11443)
Reported by: destiny6628
Patches:
     11999.patch uploaded by putnopvut (license 60)
Tested by: destiny6628, victoryure


........

................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=114210
By: Digium Subversion (svnbot) 2008-04-17 12:30:50

Repository: asterisk
Revision: 114214

_U  team/group/multiparking/
U   team/group/multiparking/CHANGES
U   team/group/multiparking/Makefile
U   team/group/multiparking/apps/app_chanspy.c
U   team/group/multiparking/apps/app_externalivr.c
U   team/group/multiparking/apps/app_festival.c
U   team/group/multiparking/apps/app_ices.c
U   team/group/multiparking/apps/app_mp3.c
U   team/group/multiparking/apps/app_nbscat.c
U   team/group/multiparking/apps/app_zapras.c
U   team/group/multiparking/channels/chan_sip.c
U   team/group/multiparking/channels/chan_zap.c
U   team/group/multiparking/configs/sip.conf.sample
U   team/group/multiparking/doc/CODING-GUIDELINES
A   team/group/multiparking/doc/chan_sip-perf-testing.txt
U   team/group/multiparking/include/asterisk/app.h
U   team/group/multiparking/include/asterisk/astobj2.h
A   team/group/multiparking/include/asterisk/dlinkedlists.h
U   team/group/multiparking/include/asterisk/dsp.h
U   team/group/multiparking/include/asterisk/frame.h
U   team/group/multiparking/include/asterisk/lock.h
U   team/group/multiparking/include/asterisk/logger.h
U   team/group/multiparking/include/asterisk/sched.h
U   team/group/multiparking/main/app.c
U   team/group/multiparking/main/asterisk.c
U   team/group/multiparking/main/astobj2.c
U   team/group/multiparking/main/dsp.c
U   team/group/multiparking/main/event.c
U   team/group/multiparking/main/frame.c
U   team/group/multiparking/main/logger.c
U   team/group/multiparking/main/sched.c
U   team/group/multiparking/main/utils.c
U   team/group/multiparking/res/res_agi.c
U   team/group/multiparking/res/res_jabber.c
U   team/group/multiparking/res/res_musiconhold.c
A   team/group/multiparking/tests/test_dlinklists.c
U   team/group/multiparking/utils/Makefile
A   team/group/multiparking/utils/refcounter.c

------------------------------------------------------------------------
r114214 | mvanbaak | 2008-04-17 12:30:47 -0500 (Thu, 17 Apr 2008) | 189 lines

Merged revisions 114172,114174-114175,114181-114183,114185,114187-114188,114190,114192,114194,114196,114199,114201-114202,114205,114208,114212 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r114172 | murf | 2008-04-16 19:14:18 +0200 (Wed, 16 Apr 2008) | 1 line

Introducing doubly linked lists to trunk from branch team/murf/bug11210.
................
r114174 | qwell | 2008-04-16 19:31:02 +0200 (Wed, 16 Apr 2008) | 14 lines

Blocked revisions 114173 via svnmerge

........
r114173 | qwell | 2008-04-16 12:30:09 -0500 (Wed, 16 Apr 2008) | 7 lines

Fix "fallthrough" behavior here, so config options in a previously configured user don't override settings in general.

(closes issue ASTERISK-11863)
Reported by: tzafrir
Patches:
     chanzap_users_sections.diff uploaded by tzafrir (license 46)

........

................
r114175 | murf | 2008-04-16 19:45:28 +0200 (Wed, 16 Apr 2008) | 1 line

Introducing various astobj2 enhancements, chief being a refcount tracing feature, and various documentation updates in astobj2.h, and the addition of standalone utility, refcounter, that will filter the trace output for unbalanced, unfreed objects. This comes from the team/murf/bug11210 branch.
................
r114181 | tilghman | 2008-04-16 22:00:27 +0200 (Wed, 16 Apr 2008) | 10 lines

Blocked revisions 114180 via svnmerge

........
r114180 | tilghman | 2008-04-16 14:59:37 -0500 (Wed, 16 Apr 2008) | 3 lines

Backport revisions for latest vpb drivers to 1.4
(Closes issue ASTERISK-11862)

........

................
r114182 | murf | 2008-04-16 22:09:39 +0200 (Wed, 16 Apr 2008) | 1 line

Introducing a small upgrade to the ast_sched_xxx  facility, to keep it from eating up lots of cpu cycles. See CHANGES. From the team/murf/bug11210 branch.
................
r114183 | murf | 2008-04-16 22:28:08 +0200 (Wed, 16 Apr 2008) | 1 line

Introducing a small optimization to event_unsubscribe; events now use a Doubly-Linked list for events, gives fast deletions, for the sake of channel driver mwi events. From team/murf/bug11210.
................
r114185 | kpfleming | 2008-04-16 22:47:30 +0200 (Wed, 16 Apr 2008) | 14 lines

Merged revisions 114184 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114184 | kpfleming | 2008-04-16 15:46:38 -0500 (Wed, 16 Apr 2008) | 6 lines

use the ZT_SET_DIALPARAMS ioctl properly by initializing the structure to all zeroes in case it contains fields that we don't write values into (which it does as of Zaptel 1.4.10)

(closes issue ASTERISK-11861)
Reported by: fnordian


........

................
r114187 | murf | 2008-04-16 22:54:41 +0200 (Wed, 16 Apr 2008) | 1 line

A small enhancement-- I added the routine log_show_lock to utils.c, which if the mentioned lock has been acquired, this routine will log to the console the normal info about that lock you'd see from the CLI when you do a 'core show locks'. It's solely for debug-- if the lock is NOT acquired, there is no output. I use it to show 'unexpected' locks, to see where/why a lock is pre-locked. This command is to be called from points of interest, like just before a trylock, and helps to spot fleeting, highly temporal locks that normally are not locked...
................
r114188 | tilghman | 2008-04-17 00:57:54 +0200 (Thu, 17 Apr 2008) | 2 lines

Standardized routines for forking processes (keeps all the specialized code in one place).

................
r114190 | murf | 2008-04-17 01:53:27 +0200 (Thu, 17 Apr 2008) | 1 line

This is the scariest commit I've done in a long time. This is the astobj2-ification of chan_sip. I've tested a number of scenarios like crazy. It used to have 4x the call setup/teardown performance of trunk, but now it's roughly at parity. I will attempt to find the bottlenecks and get it back to the 4x mark. The changes made were somewhat invasive, but the value to the community of these upgrades outweighs waiting further for more testing. Every change being made to chan_sip was lousing this code up when we tried to merge. Peers, Users, Dialogs, are all now astobj2 objects, indexed via hashtables. Refcounting is used to track objects and free them at the bitter end of their lives. Please file issues on bugs.digium.com, and PLEASE, please, please be patient. One natural advantage to all the hash-table work is that loading large sip.conf files full of thousands of peers now goes much faster. One more please: PLEASE help thrash this code and test it.
................
r114192 | seanbright | 2008-04-17 12:55:05 +0200 (Thu, 17 Apr 2008) | 9 lines

Merged revisions 114191 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114191 | seanbright | 2008-04-17 06:51:20 -0400 (Thu, 17 Apr 2008) | 1 line

Make sure we have enough room for the recording's filename.
........

................
r114194 | seanbright | 2008-04-17 14:25:23 +0200 (Thu, 17 Apr 2008) | 1 line

Update the CHANGES file with yesterday's ChanSpy change.  Sorry Kevin, just saw your e-mail.
................
r114196 | tilghman | 2008-04-17 14:59:04 +0200 (Thu, 17 Apr 2008) | 16 lines

Merged revisions 114195 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114195 | tilghman | 2008-04-17 07:56:38 -0500 (Thu, 17 Apr 2008) | 8 lines

Add special case for when the agi cannot be executed, to comply with the documentation that
we return failure in that case.
(closes issue ASTERISK-11866)
Reported by: fmueller
Patches:
      20080416__bug12462.diff.txt uploaded by Corydon76 (license 14)
Tested by: fmueller

........

................
r114199 | phsultan | 2008-04-17 15:46:17 +0200 (Thu, 17 Apr 2008) | 10 lines

Merged revisions 114198 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114198 | phsultan | 2008-04-17 15:42:23 +0200 (Thu, 17 Apr 2008) | 2 lines

Use keepalives effectively in order diagnose bug ASTERISK-11838.

........

................
r114201 | murf | 2008-04-17 16:45:16 +0200 (Thu, 17 Apr 2008) | 1 line

Thanks to snuff for finding these omissions
................
r114202 | tilghman | 2008-04-17 17:12:52 +0200 (Thu, 17 Apr 2008) | 2 lines

fileio.h does not exist; io.h does, though.

................
r114205 | russell | 2008-04-17 18:25:29 +0200 (Thu, 17 Apr 2008) | 11 lines

Merged revisions 114204 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114204 | russell | 2008-04-17 11:23:45 -0500 (Thu, 17 Apr 2008) | 3 lines

Fix the bininstall target to install from subdirs, as well.
(closes issue AST-8, patch from bmd at switchvox)

........

................
r114208 | mmichelson | 2008-04-17 18:40:12 +0200 (Thu, 17 Apr 2008) | 20 lines

Merged revisions 114207 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r114207 | mmichelson | 2008-04-17 11:28:03 -0500 (Thu, 17 Apr 2008) | 12 lines

It was possible for a reference to a frame which was part of a freed DSP to still be
referenced, leading to memory corruption and eventual crashes. This code change ensures
that the dsp is freed when we are finished with the frame. This change is very similar
to a change Russell made with translators back a month or so ago.

(closes issue ASTERISK-11443)
Reported by: destiny6628
Patches:
     11999.patch uploaded by putnopvut (license 60)
Tested by: destiny6628, victoryure


........

................
r114212 | mmichelson | 2008-04-17 18:51:09 +0200 (Thu, 17 Apr 2008) | 11 lines

Blocked revisions 114211 via svnmerge

........
r114211 | mmichelson | 2008-04-17 11:50:46 -0500 (Thu, 17 Apr 2008) | 4 lines

Add prototype for ast_dsp_frame_freed. I'm not sure how this was
compiling before...


........

................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=114214