[Home]

Summary:ASTERISK-04216: memory leak in frame/channel
Reporter:Roy Sigurd Karlsbakk (rkarlsba)Labels:
Date Opened:2005-05-18 07:19:32Date Closed:2011-06-07 14:10:35
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) leakfix.patch
( 1) queue_leak-stable.txt
Description:in ast_frdup() in frame.c, the following line is being called excessively:
   buf = malloc(len);
ast_frdup() seems to be called from channel.c:377
   f = ast_frdup(fin);
but I'm unable to trace it further back, as I really don't know the code that well.

I noticed the leak with astmm:

sipgw1*CLI> show memory summary frame.c
121071160 bytes in 437080 allocations in function 'ast_frdup' of 'frame.c'
   549168 bytes in    34 allocations in function 'ast_smoother_new' of 'frame.c'
121620328 bytes allocated 437114 units total

This seems to grow with approx 100kB per minute with the current amount of channels being about 74, about half of them being SIP, the other half being IAX2.
Comments:By: stevekstevek (stevekstevek) 2005-05-18 11:10:36

Frames are duplicated when they're queued in ast_queue_frame(), by channels..  This is normal.

If they're not being freed, you'd need to tell us exactly what apps you're running when you see this behavior.  Because if an app receives the frame, it's the apps responsibility to free the frames it gets.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-18 15:31:05

The only app I can think of, would be app_queue.
Do you know how I can trace/log this as to find where the problem is?

roy
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-22 09:24:22

Can someone please help me out and point to what can be done to find this leak? it's a rather nasty one....

roy
By: Russell Bryant (russell) 2005-05-23 19:16:45

Can you give this a try without app_queue and see if you still notice the problem?
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-23 20:08:33

it's a bit hard, since the system's in production, with quite some taffic.
I have, however, tried testing with app_queue numerous times without seeing any change.
By: Russell Bryant (russell) 2005-05-24 05:03:24

Hm ... I would really just like to verify that app_queue is the culprit before I dig into it
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-24 08:38:57

More testing shows this seems to happen when app_queue is queueing up SIP members. I have tested with a queue with Music on hold and with ringing, and no change. I have tested with one SIP member and that leaks, I also tested with one IAX2 member (forwarding to another pstn gateway) and that did not leak. The actual leak seems to be while it's ringing, although there might be some if the callie is busy, but I'm not sure yet...

To reproduce, please try the following:
- Compile asterisk with astmm
- Create a queue with an available SIP peer
- Dial into the queue
- Don't pick up, but just "show memory summary frame.c" periodically

Tested with mysqlsipfriends (as we currently use) and without, giving the same results.


By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-25 06:06:01

Bug is reproducable both on 1.0.7, latest CVS stable and CVS HEAD
By: Clod Patry (junky) 2005-05-25 06:54:22

And what about: show memory summary app_queue.c?
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-25 07:00:23

No leaks there AFAICS

sipgw1*CLI> show memory summary app_queue.c
      140 bytes in     9 allocations in function 'complete_queue' of 'app_queue.c'
     1392 bytes in     4 allocations in function 'queue_exec' of 'app_queue.c'
       38 bytes in     3 allocations in function 'ring_entry' of 'app_queue.c'
    36532 bytes in    80 allocations in function 'reload_queues' of 'app_queue.c'
38102 bytes allocated 96 units total
By: Clod Patry (junky) 2005-05-25 07:02:00

Fine,
now would you do the same thing with anything related to mysql (dont forget any cdr engine too).
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-25 07:59:36

I really don't understand why you want this info, but here:

sipgw1*CLI> show memory summary app_addon_sql_mysql.c
0 bytes allocated 0 units total
sipgw1*CLI> show memory summary cdr_addon_mysql.c
0 bytes allocated 0 units total
sipgw1*CLI>

As stated above, the same problem occurs both on CVS HEAD and -stable and with or without anything mysql-related....
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-26 11:42:37

Could this be related to ASTERISK-3891?

> NO, it's not related
>
> roy


By: Roberto Lopes (rflopes3) 2005-05-26 15:49:52

We are having the same problem on a Call Center client of us. Seems like the app_queue is the problem. Any help ?
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-27 00:58:52

I don't think the leak is in app_queue, as the leak is in frame.c:305, a malloc() in ast_frdup(). ast_frdup() is called from rtp.c:1274 in ast_rtp_write() for h261, h263, g723.1, lpc10, speex and "unknown" codecs, and I don't use any of those, so I don't think it there. Then, apart from being called a couple of times from the old chan_iax, it is only called from channel.c:377, in ast_queue_frame(). I beleive that is the source. Somewhere, ast_frframe() isn't called as it should be. This is most likely from chan_sip, as I've tried using IAX2 peers in the queue to test if the leak was reproducable with IAX, something it wasn't.

But that's as far as I get. I'm sorry to say I don't know the deep secrets of the Asterisk source.

roy
By: zamsler (zamsler) 2005-05-27 07:55:05

I have been seeing simular issues. I have * running our office phones.(8 SIP Extens, 4 ZAP FXO, broadvoice, and sixtel all using ULAW) I notice asterisk crawl to a slow at times. When I look at top, asterisk has a big chunk of memory. A reload normally fixed the problem. I believe that this happens when have have lots of incoming calls going through the call queues. I will watch a bit closer and let you know what I see.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-27 09:12:35

Are you sure the memory gets freed with a reload?
nothing happens if I do that. I got to restart the whole of asterisk to free the memory:

sipgw1:~ # ps axvf|grep -w asterisk |grep -v grep| head -1
9929 ?        S<     0:05      0   617 720402 611460 29.4  \_ asterisk -vvvg -c
sipgw1:~ # asterisk -r
Asterisk 1.0.7, Copyright (C) 1999-2004 Digium.
Written by Mark Spencer <markster@digium.com>
=========================================================================
Connected to Asterisk 1.0.7 currently running on sipgw1 (pid = 9929)
sipgw1*CLI> show memory summary frame.c
   549168 bytes in    34 allocations in function 'ast_smoother_new' of 'frame.c'
408860575 bytes in 1476035 allocations in function 'ast_frdup' of 'frame.c'
409409743 bytes allocated 1476069 units total
sipgw1*CLI> reload
sipgw1*CLI> show memory summary frame.c
   565320 bytes in    35 allocations in function 'ast_smoother_new' of 'frame.c'
408861406 bytes in 1476038 allocations in function 'ast_frdup' of 'frame.c'
409426726 bytes allocated 1476073 units total
sipgw1*CLI> exit
sipgw1:~ # ps axvf|grep -w asterisk |grep -v grep| head -1
9929 ?        S<     0:05      0   617 720402 611452 29.4  \_ asterisk -vvvg -c
sipgw1:~ #
By: zamsler (zamsler) 2005-05-27 11:51:21

I am sorry. You are correct.. I have a script that stops and starts asterisk.(ie reload) /Zac
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-29 15:49:16

I added some debug printouts, and it seems the bug is related to iax2_queue_frame() in channels/chan_iax2.c

This is with a queue with only SIP members

roy
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-29 15:54:31

This patch http://karlsbakk.net/asterisk/memory-debug.patch adds some text to see who called it. I'm not aware of any good backtracing while running the program, so I just added some text. I beleive this shows where the error's origin...
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-29 16:38:59

One addition: chan_iax2 is indeed connected to the problem. Callers dial from pstn to a pstn gateway, the pstn gateway connects to the SIP gateway using IAX2, members of the queue are connected on SIP.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-30 06:39:29

The attached patch submitted by Klaus Peter Junghanns <kpj@junghanns.net> fixes the problem.
By: Michael Jerris (mikej) 2005-05-30 09:06:15

Is the patch disclaimed?
By: Michael Jerris (mikej) 2005-05-30 09:12:47

also we need a quick code review and a few more people to try this out.  Royk, snag some people to test this out and post results to the bug.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-30 09:13:14

I don't think kapejod has ever disclaimed anything to digium.
By: Mark Spencer (markster) 2005-05-30 09:32:06

Resolved without using the attached patch.  Also removed some related dead code.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-05-30 13:22:03

This doesn't seem fixed in CVS -stable
By: Michael Jerris (mikej) 2005-05-30 14:15:17

bugs get resolved in head, then closed out when fixed in stable.  Drumkilla will commit this to stable as his time allows.
By: Russell Bryant (russell) 2005-05-30 15:28:27

Yep, mark fixed it for cvs head, but the patch will have to be re-written for 1.0 because it does not apply.
By: Clod Patry (junky) 2005-05-30 22:37:47

That should solve that issue for 1.0
By: Russell Bryant (russell) 2005-05-30 22:57:22

fixed in 1.0

I gave some karma to rkarlsba as well for promptly providing debug information throughout the report

Thanks to mark for providing the proper fix for this!
By: Clod Patry (junky) 2005-05-30 23:01:04

yes, a great thanks to RoyK for insisting and giving energy to debug that huge leak.
Thanks.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-06-08 02:55:40

Seems the patches, both the one I submitted from kapejod, and the one in CVS works fine relating to the memory leak. However, with the patches, people queue up as normal, but after a while, they _only_ get queued up, but no phones ring.
By: Roy Sigurd Karlsbakk (rkarlsba) 2005-06-08 06:04:44

This seems to be another bug.
Callers are queued up and they just stay there, no phones calling.
Please close this again

roy
By: Clod Patry (junky) 2005-06-08 06:43:26

close on reporter request.
By: Digium Subversion (svnbot) 2008-01-15 15:36:21.000-0600

Repository: asterisk
Revision: 5786

U   trunk/apps/app_queue.c

------------------------------------------------------------------------
r5786 | markster | 2008-01-15 15:36:20 -0600 (Tue, 15 Jan 2008) | 2 lines

Fix memory leak and remove useless coe (bug ASTERISK-4216, with mods)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=5786
By: Digium Subversion (svnbot) 2008-01-15 15:36:27.000-0600

Repository: asterisk
Revision: 5794

U   branches/v1-0/apps/app_queue.c

------------------------------------------------------------------------
r5794 | russell | 2008-01-15 15:36:27 -0600 (Tue, 15 Jan 2008) | 2 lines

fix mem leak (bug ASTERISK-4216)

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=5794