[Home]

Summary:ASTERISK-12746: Memory leak in channel variables
Reporter:Florent Chauveau (mav3rick)Labels:
Date Opened:2008-09-17 11:06:24Date Closed:2010-01-20 08:19:08.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Channels/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20080917__bug13505__trunk__debug.diff.txt
( 1) 20080918__bug13505.diff.txt
( 2) 20080919__bug13505.diff.txt
( 3) 20080919.alloc.txt
( 4) 20080919.summary.txt
( 5) 20081001__bug13505.diff.txt
( 6) alloc.diff.txt
( 7) outgoing.alloc.diff.txt
( 8) pj-mem-sum-alloc.zip
( 9) pj-mem-sum-alloc2.zip
(10) pj-mem-sum-alloc3.zip
(11) pj-mem-sum-alloc4.zip
Description:I detected the leak on Asterisk 1.4.21.2. I reproduced the bug easily on the SVN trunk version.

After 5000 calls, Asterisk is using more than 500Mb.

I recompiled with MALLOC_DEBUG (and DONT_OPTIMIZE), and after around 500 calls I found :

661364 bytes in 6321 allocations in file 'config.c'
(numbers increase with calls).

That was the only abnormal allocation (and the only increasing).

All 'config.c' allocations are :

XX bytes allocated in ast_variable_new at line 240 of config.c
(appeared a lot of times with different values).

I added debug to config.c to know which variable was concerned. The only variables I saw was the variables I set in call files (outgoing calls).


****** ADDITIONAL INFORMATION ******

Here is what I do.

I have an incoming call (A).
(A) is creating an outgoing call file, and sets 14 variables using "Set :". Right after the file is created, I can see in the Asterisk CLI my debug on config.c : ast_variable_new is called for each variable I set. Those variables are not freed in some case.

The outgoing call (B) is on a local channel : "Channel: Local/s@dial-context/n". The file specifies an answering context "dial-context-answer".

(A), after creating the file, is playing music (Playback).

(B) is created as a local channel, does some debug (NoOp) and then dials a SIP channel. If the call succeeds, (B) goes automatically to "dial-context-answer" (as specified in the call file).

In "dial-context-answer", (B) is doing a ChannelRedirect on (A) that is going to a MeetMe room, and then (B) goes to the same MeetMe room, using a Goto.

It's pretty complicated because it's part of a larger system. I hope there's a simpler way to reproduce it, but I did not find it.

When (A) or (B) hangs up, the other channel hangs up too, kicked by the other one off the MeetMe room.

Additional note : when I tried Asterisk trunk, I forgot dahdi-tools so I did not have the MeetMe application. In that case, there was no memory leaking. The two channels hung up while trying to "MeetMe". After compiling dahdi-tools and the MeetMe application, the leak was back...
Comments:By: pj (pj) 2008-09-17 11:30:06

I'm observing same memory leak with ast_variable_new allocations in config.c,
my Asterisk SVN-trunk-r143034 without zaptel/dahdi
after about 260 calls processed and 3days uptime:
   142183 bytes in 2296 allocations in file 'config.c'
By: Florent Chauveau (mav3rick) 2008-09-17 11:34:14

I have another box running Asterisk 1.4.21.2 with the same configuration but a much simpler Dialplan : AGI and Dial (Zap//SIP gateway). There is no memory leak is that case.
By: pj (pj) 2008-09-17 14:33:49

my asterisk is working in pure sip2sip mode, without any special things like agi and allocations still growing in config.c (even when asterisk is idle and no calls are currently processed)
asterisk -rx "memory show allocations" | grep ast_variable_new | wc -l
2317
By: Tilghman Lesher (tilghman) 2008-09-17 15:41:20

I have uploaded a patch for you to apply and work with.  Let me be clear that this patch is NOT a fix; rather, it's a change to help track down exactly where the API allocating memory is REALLY located, so we can nip it.
By: Florent Chauveau (mav3rick) 2008-09-18 02:37:31

Here are the results. After applying the patch, I don't have leaking from config.c anymore. Instead, I have :

  1322332 bytes in 12628 allocations in file 'pbx_spool.c' (increasing)

All the allocations are :

  XXX bytes allocated in       apply_outgoing at line   210 of pbx_spool.c


I've uploaded a diff of "memory show summary" with and without the patch : outgoing.alloc.diff.txt.


By: pj (pj) 2008-09-18 04:25:00

Asterisk SVN-trunk-r143405 + 20080917__bug13505__trunk__debug.diff.txt
allocation in config.c isn't growing anymore (still 10 allocations) after 22 calls processed, but instead, another modules starts leaking, mainly chan_sip, and pbx.c, mmlog is clean,
uploading alloc.diff.txt, where you can see what modules potentialy leaking

example of possible leaking in chan_sip:

       55 bytes allocated in            copy_vars at line 11725 of chan_sip.c
       63 bytes allocated in            copy_vars at line 11725 of chan_sip.c
       63 bytes allocated in            copy_vars at line 11725 of chan_sip.c
       55 bytes allocated in            copy_vars at line 11725 of chan_sip.c


By: Tilghman Lesher (tilghman) 2008-09-18 15:04:06

Okay, patch uploaded that should address both reported leaks.  pj, if you want to provide more information similar to the chan_sip.c output as to where leaks are occurring, I can take a look at those, as well.

Please keep the current debugging patch, as well as applying this new patch, so that we can verify that I found and correctly patched these issues.
By: Florent Chauveau (mav3rick) 2008-09-19 02:26:17

The patch is not working for me : Segmentation fault after 50 outgoing calls.

mmlog is full of :

WARNING: Freeing unused memory at 0x839deb8, in ast_variables_destroy of config.c, line 403
WARNING: Freeing unused memory at 0x839dfa8, in ast_variables_destroy of config.c, line 403
WARNING: Freeing unused memory at 0x83e2af8, in ast_variables_destroy of config.c, line 403
WARNING: Freeing unused memory at 0x83e2be0, in ast_variables_destroy of config.c, line 403
WARNING: Freeing unused memory at 0x825f218, in ast_variables_destroy of config.c, line 403

Ignore the last two files "after.patch1", I was not running the patched version.


By: pj (pj) 2008-09-19 03:03:01

SVN-trunk-r143405
+ 20080917__bug13505__trunk__debug.diff.txt
+ 20080918__bug13505.diff.txt
is working for me :o)
I will post mem alloc/summary after sufficient calls will be processed by my asterisk
By: Florent Chauveau (mav3rick) 2008-09-19 10:09:26

SVN-trunk-r14334 + 20080917__bug13505__trunk__debug.diff.txt

I just uploaded more memory debug output :

20080919.alloc.txt
20080919.summary.txt

dev ~ # asterisk -rx 'core show channels'
Channel              Location             State   Application(Data)
0 active channels
0 active calls
893 calls processed
By: pj (pj) 2008-09-19 10:36:04

also uploading memory summary and allocations (pj-mem-sum-alloc.zip),
asterisk with applied both patches
SVN-trunk-r143405
+ 20080917__bug13505__trunk__debug.diff.txt
+ 20080918__bug13505.diff.txt
75 calls processed
it seems, that mainly chan_sip and pbx.c is still leaking in...
copy_vars at line 11727 of chan_sip.c
add_pattern_node at line  1433 of pbx.c


By: Tilghman Lesher (tilghman) 2008-09-19 12:11:53

mav3rick:  One question:  do your outgoing calls often fail?  That's the one condition I can find that would cause a memory leak.
By: Tilghman Lesher (tilghman) 2008-09-19 12:15:28

pj: I don't think that either of those are actually leaks, unless you're running this command at a time when there are 0 channels up.
By: pj (pj) 2008-09-19 12:24:22

yes, I ran memory show commands, when asterisk was completely idle.
By: Florent Chauveau (mav3rick) 2008-09-20 01:44:02

Corydon76: yes they often fail. I'll try 20080919__bug13505.diff.txt.
By: Florent Chauveau (mav3rick) 2008-09-21 13:50:35

SVN-trunk-r14334
+ 20080917__bug13505__trunk__debug.diff.txt
+ 20080919__bug13505.diff.txt

is working perfectly. No more leaks.

Is it possible to have a patch for the 1.4 version as well ?

Thanks
By: pj (pj) 2008-09-23 06:21:04

SVN-trunk-r143737
+ 20080917__bug13505__trunk__debug.diff.txt
+ 20080919__bug13505.diff.txt
allocations still growing, mainly in chan_sip, look at
pj-mem-sum-alloc2.zip
output was taken when no call was processed, and #of peers are still about 10-12, mmlog is clean

core show uptime
System uptime: 3 days, 2 hours, 11 minutes, 7 seconds
Last reload: 3 days, 2 hours, 11 minutes, 7 seconds

core show channels count
0 active channels
0 active calls
124 calls processed
By: Tilghman Lesher (tilghman) 2008-10-01 13:43:03

pj:  well, I found a few more memory leaks, though I still haven't found where chan_sip is leaking.  Next time this happens, though, please show me the output of 'sip show channels'.
By: pj (pj) 2008-10-01 13:47:02

nothing in special from 'sip show channels' output, no death sip channels
By: Tilghman Lesher (tilghman) 2008-10-01 13:48:54

Oh, and... new patch uploaded.
By: pj (pj) 2008-10-04 14:21:29

SVN-trunk-r145692 + 20081001__bug13505.diff.txt
chan_sip is still leaking, other modules are almost fine
uploading pj-mem-sum-alloc3.zip, you can see memory allocations difference after asterisk startup and after two days of asterisk runtime.
Can I help with some other debugging, eg. valgrind?
By: Tilghman Lesher (tilghman) 2008-10-06 15:09:12

pj: are you sure you had the latest patch applied?  There are several messages in your logs that quite simply cannot occur with the latest patch.
By: pj (pj) 2008-10-07 00:22:20

yes, I have your latest patch (size ~11940B), but applied alone, because it failed to apply, when I tried apply to asterisk patched with your first 20080917__bug13505__trunk__debug.diff.txt
p.s. I'm applying all patches to fresh downloaded asterisk trunk
By: Tilghman Lesher (tilghman) 2008-10-07 11:48:16

pj: the latest patch is comprehensive.  You do not need to apply any other patches first.
By: pj (pj) 2008-10-07 12:41:17

OK, so that I have your latest patch applied correctly to asterisk trunk.
By: David Brillert (aragon) 2008-10-09 22:17:20

pj any news?

I'm hoping everything is OK so Corydon76 can commit to branch...
By: pj (pj) 2008-10-10 02:15:22

uploading another set (pj-mem-sum-alloc4.zip) of memory summary/memory allocations, that showing, that my asterisk still have huge (in comparison, to relativelly few call processed) memory leaks in chan_sip.
it seems, that something is still wrong with:
copy_vars at line 11743 of chan_sip.c
can't be it related to another my issue ASTERISK-1303115 ? from time to time, I have "could NOT get the channel lock" msg on CLI (mainly during hangup, so due to this, variables aren't correctly destroyed?) but I haven't any death sip channels.
Also notice, that I have relatively slow cpu (mobile PII, 300MHz).


By: Tilghman Lesher (tilghman) 2008-10-14 17:21:25

pj: the additional dumps are not helpful.  I am closing out this issue, as it solves the issue for the original reporter.  You are welcome to open a new issue with your output, but additional research is needed as to where chan_sip is actually leaking.  We can all see that it is leaking, but the problem is that it's not easy to figure out where the memory should be freed (or rather why the memory isn't being freed where it should be).
By: Digium Subversion (svnbot) 2008-10-14 17:27:42

Repository: asterisk
Revision: 149199

U   trunk/channels/chan_sip.c
U   trunk/include/asterisk/chanvars.h
U   trunk/include/asterisk/config.h
U   trunk/include/asterisk/hashtab.h
U   trunk/include/asterisk/strings.h
U   trunk/main/chanvars.c
U   trunk/main/config.c
U   trunk/main/hashtab.c
U   trunk/pbx/pbx_spool.c
U   trunk/res/res_indications.c

------------------------------------------------------------------------
r149199 | tilghman | 2008-10-14 17:27:42 -0500 (Tue, 14 Oct 2008) | 8 lines

Add additional memory debugging to several core APIs, and fix several memory
leaks found with these changes.
(Closes issue ASTERISK-12746, closes issue ASTERISK-12765)
Reported by: mav3rick, triccyx
Patches:
      20081001__bug13505.diff.txt uploaded by Corydon76 (license 14)
Tested by: mav3rick, triccyx

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=149199