[Home]

Summary:ASTERISK-15404: [patch] Huge memory leak
Reporter:Florent Chauveau (mav3rick)Labels:
Date Opened:2010-01-06 03:16:04.000-0600Date Closed:2010-01-20 08:19:08.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Applications/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 4.summary.txt
( 1) 4.valgrind.txt
( 2) 5.summary.txt
( 3) 5.valgrind.txt
( 4) 6.valgrind.txt
( 5) allocations.txt.gz
( 6) issue16554_20100118.patch
( 7) issue16554_20100119.patch
( 8) summary.2.txt
( 9) summary.3.txt
(10) summary.txt
(11) valgrind.3.txt
Description:I detected a memory leak on all my Asterisk servers.

After 1 day, Asterisk is using 100+ meg.

I recompiled with MALLOC_DEBUG on DONT_OPTIMIZE.
See dump of memory summary and memory allocations.
Comments:By: Florent Chauveau (mav3rick) 2010-01-07 09:28:59.000-0600

Uploaded a new memory summary.
I suppose I need a patch to see where the ast_variable_new are coming from.
By: Sean Bright (seanbright) 2010-01-11 16:48:44.000-0600

If possible, could you follow the instructions in doc/valgrind.txt so we can see where the memory is being allocated?
By: Florent Chauveau (mav3rick) 2010-01-12 10:53:16.000-0600

I just uploaded a new memory summary with valgrind debug
By: Sean Bright (seanbright) 2010-01-12 11:20:22.000-0600

The valgrind output is very short.  Did you upload the entire file?  There should be much more detail in that file based on the memory summary you uploaded.
By: Florent Chauveau (mav3rick) 2010-01-12 11:35:50.000-0600

Yes it's the entire file. I'm surprised too. I can try again. I use valgrind's Debian, with this .supp file : https://issues.asterisk.org/file_download.php?file_id=24676&type=bug

the one included in Asterisk outputs : FATAL: in suppressions file '/root/pbx-install/asterisk-1.4.28/contrib/valgrind.supp': location should start with 'fun:' or 'obj:'

maybe I can try without the supp file.
By: Sean Bright (seanbright) 2010-01-12 11:38:44.000-0600

Sure.  Try without the suppression file.

Also, what are you doing on this server?  Lots of MeetMe conferences?  Asterisk Manager calls?
By: Florent Chauveau (mav3rick) 2010-01-12 12:21:33.000-0600

Lots of things.

MeetMe, outgoing calls (.call files), AMI.
By: Florent Chauveau (mav3rick) 2010-01-13 07:18:23.000-0600

Uploaded new summary and valgrind.

I suppose I re-run with --leak-check=full ?
By: Sean Bright (seanbright) 2010-01-13 08:21:56.000-0600

Yes sir.  If you don't mind.
By: Florent Chauveau (mav3rick) 2010-01-13 09:58:50.000-0600

Okay, I will let it run for 1 day and send the results.
By: Sean Bright (seanbright) 2010-01-15 10:50:55.000-0600

Do you have an updated valgrind dump?
By: Florent Chauveau (mav3rick) 2010-01-15 12:20:30.000-0600

Here you go.

The memory seems to be growing from here :

 47806714 bytes in 866197 allocations in file 'config.c'

However I don't see anything relevant in valgrind, maybe you will :)
By: Sean Bright (seanbright) 2010-01-15 12:24:07.000-0600

Thank you.  Please re-run and add these flags:

   --leak-check=full --show-reachable=yes

And based on how much memory is getting allocated, I don't think we need a full day's worth of data.  A few hours would probably be sufficient.
By: Florent Chauveau (mav3rick) 2010-01-15 12:32:47.000-0600

Sure. I'll upload the results in a few hours.
By: Florent Chauveau (mav3rick) 2010-01-17 07:15:22.000-0600

Sorry I forgot to upload the file and to do the memory summary.

Hope this helps.

Asterisk was using more than 200mb when I stopped it.
By: Florent Chauveau (mav3rick) 2010-01-17 07:18:33.000-0600

Note : I have the same box (same Debian) with the same dialplan and configuration on Asterisk 1.4.21.2, and there is no memory leak.

Hope this helps.
By: Sean Bright (seanbright) 2010-01-18 05:36:07.000-0600

That valgrind output isn't helpful.  I spent some time this weekend setting up a test environment and wasn't able to duplicate the leak.  Will continue testing.

<b>Edit:</b> I've narrowed it down.  Working on a patch now.


By: Sean Bright (seanbright) 2010-01-18 09:17:57.000-0600

Patch uploaded.  Please test and report results.  Thanks.
By: Florent Chauveau (mav3rick) 2010-01-18 09:27:40.000-0600

Thanks. Currently testing. I'll come back in a few hours with results.
By: Florent Chauveau (mav3rick) 2010-01-18 10:34:02.000-0600

From what I can see, there is still a leak in config.c

Random checks :

   722937 bytes in 13318 allocations in file 'config.c'
  2596962 bytes in 47573 allocations in file 'config.c'
  2838198 bytes in 52066 allocations in file 'config.c'
  3285482 bytes in 60270 allocations in file 'config.c'
  4127653 bytes in 75688 allocations in file 'config.c'

Do you need a full memory allocations or summary ?
By: Sean Bright (seanbright) 2010-01-18 10:35:57.000-0600

The patch attached addresses a leak in the GetConfig manager action.  Are there specific manager actions that you are executing in a recurring manner?
By: Florent Chauveau (mav3rick) 2010-01-18 10:49:59.000-0600

Yes, I have a daemon connecting to all my boxes that sends :

ACTION: LOGIN

and then,

ACTION: COMMAND with "core show channels"

that's it.
By: Sean Bright (seanbright) 2010-01-18 10:58:19.000-0600

Is this a production machine?  Or would it be possible for me to get remote access to it to be able to test/debug?
By: Florent Chauveau (mav3rick) 2010-01-18 11:15:00.000-0600

What is your email address ?
By: Digium Subversion (svnbot) 2010-01-18 13:54:20.000-0600

Repository: asterisk
Revision: 241015

U   branches/1.4/main/config.c

------------------------------------------------------------------------
r241015 | seanbright | 2010-01-18 13:54:19 -0600 (Mon, 18 Jan 2010) | 12 lines

Plug a memory leak when reading configs with their comments.

While reading through configuration files with the intent of returning their
full contents (comments specifically) we allocated some memory and then forgot
to free it.  This doesn't fix 16554 but clears up a leak I had in the lab.

(issue ASTERISK-15404)
Reported by: mav3rick
Patches:
     issue16554_20100118.patch uploaded by seanbright (license 71)
Tested by: seanbright

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241015
By: Digium Subversion (svnbot) 2010-01-18 13:57:53.000-0600

Repository: asterisk
Revision: 241016

_U  trunk/
U   trunk/main/config.c

------------------------------------------------------------------------
r241016 | seanbright | 2010-01-18 13:57:53 -0600 (Mon, 18 Jan 2010) | 19 lines

Merged revisions 241015 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r241015 | seanbright | 2010-01-18 14:54:19 -0500 (Mon, 18 Jan 2010) | 12 lines
 
 Plug a memory leak when reading configs with their comments.
 
 While reading through configuration files with the intent of returning their
 full contents (comments specifically) we allocated some memory and then forgot
 to free it.  This doesn't fix 16554 but clears up a leak I had in the lab.
 
 (issue ASTERISK-15404)
 Reported by: mav3rick
 Patches:
       issue16554_20100118.patch uploaded by seanbright (license 71)
 Tested by: seanbright
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241016
By: Digium Subversion (svnbot) 2010-01-18 14:01:47.000-0600

Repository: asterisk
Revision: 241017

_U  branches/1.6.0/
U   branches/1.6.0/main/config.c

------------------------------------------------------------------------
r241017 | seanbright | 2010-01-18 14:01:46 -0600 (Mon, 18 Jan 2010) | 26 lines

Merged revisions 241016 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r241016 | seanbright | 2010-01-18 14:57:52 -0500 (Mon, 18 Jan 2010) | 19 lines
 
 Merged revisions 241015 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r241015 | seanbright | 2010-01-18 14:54:19 -0500 (Mon, 18 Jan 2010) | 12 lines
   
   Plug a memory leak when reading configs with their comments.
   
   While reading through configuration files with the intent of returning their
   full contents (comments specifically) we allocated some memory and then forgot
   to free it.  This doesn't fix 16554 but clears up a leak I had in the lab.
   
   (issue ASTERISK-15404)
   Reported by: mav3rick
   Patches:
         issue16554_20100118.patch uploaded by seanbright (license 71)
   Tested by: seanbright
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241017
By: Digium Subversion (svnbot) 2010-01-18 14:01:51.000-0600

Repository: asterisk
Revision: 241018

_U  branches/1.6.1/
U   branches/1.6.1/main/config.c

------------------------------------------------------------------------
r241018 | seanbright | 2010-01-18 14:01:50 -0600 (Mon, 18 Jan 2010) | 26 lines

Merged revisions 241016 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r241016 | seanbright | 2010-01-18 14:57:52 -0500 (Mon, 18 Jan 2010) | 19 lines
 
 Merged revisions 241015 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r241015 | seanbright | 2010-01-18 14:54:19 -0500 (Mon, 18 Jan 2010) | 12 lines
   
   Plug a memory leak when reading configs with their comments.
   
   While reading through configuration files with the intent of returning their
   full contents (comments specifically) we allocated some memory and then forgot
   to free it.  This doesn't fix 16554 but clears up a leak I had in the lab.
   
   (issue ASTERISK-15404)
   Reported by: mav3rick
   Patches:
         issue16554_20100118.patch uploaded by seanbright (license 71)
   Tested by: seanbright
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241018
By: Digium Subversion (svnbot) 2010-01-18 14:01:55.000-0600

Repository: asterisk
Revision: 241019

_U  branches/1.6.2/
U   branches/1.6.2/main/config.c

------------------------------------------------------------------------
r241019 | seanbright | 2010-01-18 14:01:54 -0600 (Mon, 18 Jan 2010) | 26 lines

Merged revisions 241016 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r241016 | seanbright | 2010-01-18 14:57:52 -0500 (Mon, 18 Jan 2010) | 19 lines
 
 Merged revisions 241015 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r241015 | seanbright | 2010-01-18 14:54:19 -0500 (Mon, 18 Jan 2010) | 12 lines
   
   Plug a memory leak when reading configs with their comments.
   
   While reading through configuration files with the intent of returning their
   full contents (comments specifically) we allocated some memory and then forgot
   to free it.  This doesn't fix 16554 but clears up a leak I had in the lab.
   
   (issue ASTERISK-15404)
   Reported by: mav3rick
   Patches:
         issue16554_20100118.patch uploaded by seanbright (license 71)
   Tested by: seanbright
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241019
By: Sean Bright (seanbright) 2010-01-19 13:37:06.000-0600

The most recent patch is for another leak in pbx_spool (the module that handles call files).  This patch has already been applied to your install.
By: Florent Chauveau (mav3rick) 2010-01-20 04:52:37.000-0600

It's perfect. After 12 hours, there is no leak at all.

[...]
     1423 bytes in 26 allocations in file 'config.c'
3076201 bytes allocated (13344 in caches) in 67673 allocations

Thank you.

Will those patches work against Asterisk 1.6.2 ? I have some boxes running Asterisk 1.6 that might be affected.

Regards,
By: Sean Bright (seanbright) 2010-01-20 08:02:05.000-0600

Awesome.

As for trunk and the 1.6.x branches, I will fix the leak in those branches if they exhibit the problem.  Thanks for all the help tracking this down.
By: Digium Subversion (svnbot) 2010-01-20 08:06:51.000-0600

Repository: asterisk
Revision: 241543

U   branches/1.4/pbx/pbx_spool.c

------------------------------------------------------------------------
r241543 | seanbright | 2010-01-20 08:06:50 -0600 (Wed, 20 Jan 2010) | 14 lines

Fix a memory leak in pbx_spool when using SetVar in a call file.

In pbx_spool, when we are freeing our 'outgoing' struct, we weren't deallocating
the ast_variable list we had built from SetVars in a call file.  Adding a call to
ast_variables_destroy in our deallocation routine works, but only if the variables
have not already been passed into ast_pbx_outgoing_app() or _exten(), both of
which take care of destroying the variable list for us.

(closes issue ASTERISK-15404)
Reported by: mav3rick
Patches:
     issue16554_20100119.patch uploaded by seanbright (license 71)
Tested by: mav3rick

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241543
By: Digium Subversion (svnbot) 2010-01-20 08:14:54.000-0600

Repository: asterisk
Revision: 241544

U   branches/1.4/pbx/pbx_spool.c

------------------------------------------------------------------------
r241544 | seanbright | 2010-01-20 08:12:45 -0600 (Wed, 20 Jan 2010) | 9 lines

Modify fix for issue 16554 to be more inline with what is already in trunk.

I should have taken a closer look at trunk/1.6.x, as this bug has already been
fixed in a much more simple manner, by just settings o->vars to NULL after the
ast_pbx_outgoing_* calls.

(issue ASTERISK-15404)
Reported by: mav3rick

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=241544
By: Sean Bright (seanbright) 2010-01-20 08:18:46.000-0600

mav3rick!  You're killing me, you already had this bug fixed for you in issue ASTERISK-12746.  Ah well.