[Home]

Summary:ASTERISK-05260: [patch] usage of call files for outbound calls causes memory leak
Reporter:Bruno Voigt (bvoigt)Labels:
Date Opened:2005-10-07 08:14:38Date Closed:2011-06-07 14:02:53
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20051011__bug5406.diff.txt
( 1) boxa-outbound-cfg.txt
( 2) bt.txt
Description:I've detected a reproducible severe memory leak when using call files to generate outbound calls .
This forces me to restart asterisk at least every 6 hours.

For loadtests asterisk box A equipped with a TE410P (EuroISDN cpe) is connected via span 1
to another similar asterisk box B acting as EuroISDN master.

I'm performing load tests by contiously feeding up to concurrent 30 call files to /var/spool/asterisk/outgoing/ on box A
which inititate via a dialplan context/extension a outbound call (redirected via chan_local) to box b
playing some preexisting wavfiles followed by hangup.

Using Asterisk CVS HEAD of 20051007 (also verified with 1.20-beta1):
I see a memory growth for the asterisk process from 29M VIRT / 9M RES to 238M VIRT / 211M RES after 25000 completed calls.

Using Asterisk 1.0.9 (also verified with Asterisk 1.0.9-BRIstuffed-0.2.0-RC8o):
I see a memory growth for the asterisk process from 39M VIRT / 11M RES to 179M VIRT / 22M RES after 1000 completed calls.

The box B acepting/recording the calls doesnt show such a memory growth over time.



****** ADDITIONAL INFORMATION ******

The dial construct via chan_local is there to have some additional information about the call from the callgenerator
to be tunneld thru asterisk and to provide direct feedback about the call success/failure to the callfile generator process.
The test-dial context also allows to use SetCallerPres() before dealing.
Comments:By: Tilghman Lesher (tilghman) 2005-10-07 09:16:20

Given that you're using so many different pieces, I'd like to see this isolated down to call-files, instead of one of the many other pieces in here.

For example, instead of using the Local channel, could you use an IAX2 channel, then call a context/extension/priority that does nothing but Hangup() ?  This would eliminate both the Local channel, as well as the miscellaneous CDR applications as possible causes of the memory leak.

I'm concerned primarily because I've already checked the pbx_spool code and cannot find an error which would cause a memory leak, unless there is a bug in the threading implementation.
By: Bruno Voigt (bvoigt) 2005-10-11 09:10:07

I removed the usage of chan_local.
I stripped the generated callfiles to:

Channel: IAX2/pbxtest/01234567890
Context: test
Extension: s
Priority: 1
CallerId: 0123456789
MaxRetries: 0
WaitTime: 35
RetryTime: 10
Account: TEST

used context of extensions.conf reduced to:

[test]
exten => s,1,Playback(/test/test1)
exten => s,n,Playback(/test/test2)
exten => s,n,Hangup

test1 and test2 are .wav-files in SOX-format "-sw -r 8000 -c1".

Regardless of using IAX2 or ZAP a continous memory growth is seen over time.

Channel: IAX2/pbxtest/01234567890
Channel: ZAP/r1/01234567890
By: Tilghman Lesher (tilghman) 2005-10-11 14:59:15

Okay, I found one memory leak.  If you remove the ACCOUNT=test line from your config, do you still get this memory leak?  If so, then there's more than one leak.  If not, then I have the patch for it.
By: Mark Spencer (markster) 2005-10-11 16:49:50

Fixed in CVS head, with mods, thanks!
By: Russell Bryant (russell) 2005-10-11 17:17:33

this patch only affects code that is present in cvs head.  If there is a memory leak in 1.0.9, then it is likely still there ...
By: Bruno Voigt (bvoigt) 2005-10-12 03:36:39

Thanks for providing the patch, but its seems to have potential for improvement :-)
If I now use current CVS HEAD my initial callfile with Account and SetVar statements then asterisk crashes upon completion of a call.
By: Tilghman Lesher (tilghman) 2005-10-12 09:07:13

If you revert Mark's change and apply my patch as attached above, there will not be a problem.
By: Mark Spencer (markster) 2005-10-12 11:01:05

The two patches are functionally identical.  Can you provide a backtrace of your new segfault?
By: Bruno Voigt (bvoigt) 2005-10-12 11:54:41

What are the steps to produce a backtrace?
I have currently the normally compiled CVS HEAD from today.

callfile:

Channel: IAX2/pbxtest/041095559012345
Context: test
Extension: s
Priority: 1
CallerId: 01722270201
MaxRetries: 0
WaitTime: 35
Account: TEST

[test]
exten => s,1,Hangup

Connected to Asterisk CVS HEAD currently running on pbx-mr (pid = 9464)
pbx-mr*CLI> set debug 9
Core debug was 0 and is now 9
pbx-mr*CLI> set verbose 9
Verbosity was 0 and is now 9
   -- Attempting call on IAX2/pbxtest/041095559012345 for s@test:1 (Retry 1)
   -- Call accepted by 172.17.94.121 (format slin)
   -- Format for call is slin
pbx-mr*CLI>
Disconnected from Asterisk server

Without the Account: TEST line it doesnt crash!
By: Tilghman Lesher (tilghman) 2005-10-12 12:09:41

There are instructions for generating a backtrace here:

http://www.voip-info.org/tiki-index.php?page=Asterisk%20debugging
By: Clod Patry (junky) 2005-10-12 12:30:22

or simply read README.backtrace probably in /usr/src/asterisk/doc/
By: Bruno Voigt (bvoigt) 2005-10-12 12:50:09

Ok, thanks for the backtrace links. here it is: bt.txt

BTW the system is a debian 3.1/testing on Compaq/HP DL380G2 P3 1100Mhz
ii  libgnutls11    1.0.16-13.1        GNU TLS library - runtime library
ii  libc6          2.3.5-6            GNU C Library: Shared libraries and Timezone data
ii  gcc            3.3.5-3            The GNU C compiler
By: Tilghman Lesher (tilghman) 2005-10-12 15:14:22

Also, remove the Account line, run the battery of tests again, and see if you still have the memory leak.  If we're chasing down the wrong road for the memory leak, I'd like to know that before we go any further.
By: Bruno Voigt (bvoigt) 2005-10-12 15:45:53

without the Account: and SetVar: lines in the callfile,
using a context with Playback()+Hangup() I see no memory growth.
By: Mark Spencer (markster) 2005-10-12 18:07:37

I cannot duplicate this memory leak.  Are you confirming it's really there with enabling malloc debugging?  With both malloc debugging *and* looking at ps auxww i can't duplicate the leak.  Can you send the minimum call file / extension that creates it?
By: Bruno Voigt (bvoigt) 2005-10-17 08:45:40

Hi all, on 20051014 I decided to update my system to the lastet debian/testing snapshot (gcc34 instead of gcc33, new libc6 etc).
I also got the then current asterisk CVS head.

Since then I no longer see the massive memory usage of asterisk as before (going from 29M to 238M after 25000 calls).

So I suspect some buggy debian testing lib to contribute the largest part to my problem, if it isnt to some changes from asterisk CVS head from 20051012 to 20051014.

Although I still see some memory growth over time, but on a much smaller scale.

I wasnt able yet to nail it down for sure to one asterisk feature used by my (attached boxa-outbound-cfg.txt) dialplan / callfile, e.g. usage of chan_local etc.

Currently I can live with the situation, but will continue trying to create a short as possible selfcontained example (and will look forward to have it veryified on debian/stable before submitting another bugreport...)

So I must thank you a lot for your patience and help in analyzing.
I suggest to close this ticket for now.

bye, Bruno
By: Tilghman Lesher (tilghman) 2005-10-17 11:24:05

Closed by request of reporter
By: Digium Subversion (svnbot) 2008-01-15 15:50:27.000-0600

Repository: asterisk
Revision: 6741

U   trunk/pbx/pbx_spool.c

------------------------------------------------------------------------
r6741 | markster | 2008-01-15 15:50:27 -0600 (Tue, 15 Jan 2008) | 2 lines

Fix memory leak in outbound calls (bug ASTERISK-5260, with mods)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=6741