[Home]

Summary:ASTERISK-00953: memory leak
Reporter:jcs (jcs)Labels:
Date Opened:2004-01-30 10:00:01.000-0600Date Closed:2004-09-25 02:55:35
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:asterisk leaks memory to the point where it has to be restarted

****** ADDITIONAL INFORMATION ******

we have two identical servers running asterisk 0.7.0 on openbsd.  the primary server eventually holds about 70 registered sip users and starts out with about 10 megs of memory usage.  over the course of 3 days, it increases to about 50 megs and will continue to increase, whereas the second server which does not take much traffic is still at about 10 megs of memory.

dumps from 'show memory summary' and 'show memory allocations' from each day are at http://twofish.eng.dls.net/~jcs/asterisk_memory.tar.gz
Comments:By: Mark Spencer (markster) 2004-02-01 11:21:07.000-0600

I don't see anything very exciting there.  Can we try to use valgrind to find the allocations (without the asterisk memory debugging)?
By: Olle Johansson (oej) 2004-02-01 12:25:39.000-0600

valgrind is Linux only.
By: sdolloff (sdolloff) 2004-02-02 00:08:39.000-0600

Jan 31 10:35:05 WARNING[1011849728]: Unable to create RTP session: Cannot alloca
te memory
Jan 31 10:35:05 WARNING[1011849728]: Unable to build sip pvt data for MWI
Jan 31 10:35:05 WARNING[1011849728]: Unable to create RTP session: Cannot alloca
te memory

Brief snippet from /var/logs/asterisk/messages.  Thousands of similar messages.
By: sdolloff (sdolloff) 2004-02-02 17:51:38.000-0600

OK, this memory leak can also be produced by running reload from the console.  Can other people please see if this is reproducable for them?  This was done with today's CVS.

I am going to include a simple modules.conf file, extensions.conf and sip.conf file that I was able to reproduce the problem with.

;
; Module Loader configuration file
;

[modules]
autoload=no
noload => pbx_wilcalu.so
load => pbx_spool.so
load => pbx_config.so
noload => format_wav_gsm.so
load => format_wav.so
noload => format_vox.so
noload => format_pcm_alaw.so
noload => format_pcm.so
noload => format_jpeg.so
noload => format_h263.so
noload => format_gsm.so
noload => format_g729.so
load => codec_ulaw.so
noload => codec_lpc10.so
noload => codec_ilbc.so
noload => codec_gsm.so
noload => codec_alaw.so
noload => codec_adpcm.so
noload => codec_a_mu.so
noload => chan_skinny.so
load => chan_sip.so
noload => chan_modem_i4l.so
noload => chan_modem_bestdata.so
noload => chan_mgcp.so
noload => chan_local.so
load => chan_iax2.so
noload => chan_iax.so
noload => chan_agent.so
load => cdr_csv.so
noload => app_zapateller.so
noload => app_waitforring.so
load => app_voicemail.so
noload => app_url.soi
noload => app_transfer.so
noload => app_system.so
noload => app_substring.so
noload => app_striplsd.so
noload => app_softhangup.so
noload => app_setcidnum.so
noload => app_setcidname.so
noload => app_setcdruserfield.so
noload => app_setcallerid.so
noload => app_senddtmf.so
noload => app_sayunixtime.so
noload => app_record.so
noload => app_read.so
noload => app_random.so
noload => app_queue.so
noload => app_qcall.so
noload => app_privacy.so
load => app_playback.so
noload => app_parkandanounce.so
noload => app_mp3.so
noload => app_milliwatt.so
noload => app_lookupcidname.so
noload => app_macro.so
noload => app_lookupblacklist.so
noload => app_image.so
load => app_hasnewvoicemail.so
noload => app_getcpeid.so
noload => app_festival.so
noload => app_enumlookup.so
noload => app_echo.so
noload => app_disa.so
noload => app_directory.so
load => app_dial.so
noload => app_db.so
load => app_datetime.so
noload => app_cut.so
load => app_chanisavail.so
load => app_cdr.so
load => app_authenticate.so
load => app_agi.so
noload => app_adsiprog.so
load => res_parking.so
load => res_musiconhold.so
noload => res_monitor.so
noload => res_indications.so
load => res_crypto.so
load => res_adsi.so
noload => chan_modem_aopen.so
noload => chan_modem.so
;
noload => pbx_gtkconsole.so
noload => pbx_kdeconsole.so
noload => app_intercom.so
;
noload => chan_alsa.so
noload => chan_oss.so
;
[global]
;chan_modem.so=yes

----------------------------------------------------------------
;sip.conf

[general]
port = 5060                     ; Port to bind to
context = default               ; Default for incoming calls
disallow=all
allow=ulaw
dtmfmode=rfc2833

[VGW01]
type=friend
nat=no
host=209.242.1.1
context=default

[VGW02]
type=friend
nat=no
host=209.242.1.2
context=default

[8477190222]
type=friend
secret=1234
nat=yes
host=dynamic
canreinvite=no
qualify=yes
mailbox=8477190222
context=unlimited

---------------------------------------

;extensions.conf
[globals]

[default]
include => on-net

[unlimited]
include => test

[on-net]
exten => _NXXNXXXXXX,1,NoOp
exten => _1NXXNXXXXXX,1,StripMSD(1)
exten => 8477190222,2,Dial(SIP/${EXTEN},24,r)
exten => 8477190222,3,Dial(IAX2/voip1u@voip2p/${EXTEN},25,r)
exten => 8477190222,4,Voicemail(u${EXTEN})
exten => 8477190222,5,Hangup


[test]
exten => 18478544799,1,Dial(SIP/${EXTEN}@VGW01)
exten => 18478544799,2,Dial(SIP/${EXTEN}@VGW02)

edited on: 02-03-04 09:38
By: sdolloff (sdolloff) 2004-02-06 14:58:09.000-0600

I just loaded rh9 and loaded asterisk and was able to reproduce the bug by doing a reload.  The memory usage goes up every single time.  I have a hard time understanding why no-one else would be able to reproduce this.
By: Brian West (bkw918) 2004-02-06 23:16:45.000-0600

ok do this.. get gdb, valgrind installed and find kram or myself in #asterisk (bkw_ or kram) we want to watch this happen with your configs.  

bkw
By: sdolloff (sdolloff) 2004-02-06 23:57:16.000-0600

I will get them installed on monday, but I was able to reproduce with the default configs.  just type reload on the console and watch the mem usage go up.
By: sdolloff (sdolloff) 2004-02-09 16:40:35.000-0600

OK, I have gdb and valgrind installed on the system.  If you can post the commands that you would like run at different points, I might be able to get the info to you.  If you would like to look at the system live, I can be reached at sdolloff[at]noc.dls.net or 847-854-4799x256 otherwise I will try again tomorrow.  If I am on #asterisk it's as vardalak.
By: Mark Spencer (markster) 2004-02-10 21:52:53.000-0600

I believe I've fixed the leak.
By: jcs (jcs) 2004-02-10 22:48:01.000-0600

i built the latest cvs tree and the memory usage still increases with each reload, but it's only like 20 bytes now.

old code, prior to today, each line after a reload:
root     22427  8.0  6.2  5012  8132 pa  S+    10:25PM    0:00.77 /usr/loc
root     22427  4.8  7.3  6428  9552 pa  S+    10:25PM    0:04.96 /usr/loc
root     22427  6.0  8.2  7600 10724 pa  S+    10:25PM    0:08.81 /usr/loc
root     22427 13.3  9.1  8788 11920 pa  S+    10:25PM    0:12.60 /usr/loc
root     22427 18.0 10.3 10364 13496 pa  S+    10:25PM    0:17.71 /usr/loc

new code with your fixes, each line after a reload:
root     15510  2.3  4.5  5040  5936 pa  SX+   10:38PM    0:00.81 /usr/loc
root     15510  1.6  4.7  5184  6120 pa  SX+   10:38PM    0:01.10 /usr/loc
root     15510 12.7  4.3  5220  5636 pa  SX+   10:38PM    0:04.65 /usr/loc
root     15510 17.8  4.3  5232  5648 pa  SX+   10:38PM    0:08.44 /usr/loc
root     15510 20.2  4.3  5244  5660 pa  SX+   10:38PM    0:12.74 /usr/loc
root     15510 13.9  4.4  5268  5684 pa  SX+   10:38PM    0:16.58 /usr/loc
root     15510  4.2  4.4  5284  5700 pa  SX+   10:38PM    0:20.93 /usr/loc
root     15510 16.7  4.4  5304  5720 pa  SX+   10:38PM    0:25.76 /usr/loc

by the way, a reload with the latest code resulted in:

asterisk in free(): error: chunk is already free

Program received signal SIGABRT, Aborted.
0x1deffcd in _thread_sys_kill ()
(gdb) where
#0  0x1deffcd in _thread_sys_kill ()
#1  0x1e1ba85 in abort ()
#2  0x1df2c01 in _thread_sys_execve ()
#3  0x1df2c37 in _thread_sys_execve ()
#4  0x1df3af7 in _thread_sys_execve ()
ASTERISK-1  0x1df3d6b in free ()
ASTERISK-2  0x1c03f479 in ast_unregister_indication_country (country=0x0)
   at indications.c:321
ASTERISK-3  0x5d53d19 in reload () at res_indications.c:392
ASTERISK-4  0x1c00d18e in ast_module_reload () at loader.c:169
ASTERISK-5  0x1c0233a9 in handle_reload (fd=15, argc=1, argv=0x3c4b3adc) at cli.c:106
ASTERISK-6 0x1c0231a9 in ast_cli_command (fd=15, s=0x3c4b3cb8 "reload") at cli.c:1007
ASTERISK-7 0x1c03648c in netconsole (vconsole=0x4012) at asterisk.c:214
ASTERISK-8 0x3f8bcbd in _thread_start ()
ASTERISK-9 0x1f in ?? ()
Error accessing memory address 0xffffffff: Invalid argument.
(gdb) quit

i commted out indications.c:321 when doing my tests with the new code.
By: jcs (jcs) 2004-02-10 22:51:09.000-0600

sorry, i meant to say each 'ps' line above was after 15 reloads, so it's now leaking ~20 bytes after 15 reloads.
By: Mark Spencer (markster) 2004-02-11 13:13:16.000-0600

I don't trust "ps" I trust valgrind.  If valgrind can find the leak then I can trust it.
By: Brian West (bkw918) 2004-02-11 23:22:08.000-0600

jcs: does it leak when run under valgrind?
By: Brian West (bkw918) 2004-02-14 23:07:27.000-0600

Fixed.  If you find that its leaking under valgrind again please reopen.