[Home]

Summary:ASTERISK-17301: Memory Leak in chan_sip.c with Asterisk Version 1.8.2.x & 1.8.3.x
Reporter:Thomas Z. (zircon)Labels:
Date Opened:2011-01-28 02:51:45.000-0600Date Closed:2011-11-30 12:34:14.000-0600
Priority:MajorRegression?
Status:Closed/CompleteComponents:Channels/chan_sip/DatabaseSupport
Versions:1.8.2 1.8.3 Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) memory_allocations_-_uptime_1d_20h.txt.tar.gz
Description:Hello,

I think there is a critical memory leak in the current version.

We must restart Asterisk every 1 - 2 days because the memory is over 600 MB of RAM.

Also if there is no calls done in this time.

Yesturday I installed the 1.8.2.3 Version with memory debugging.

Attached you will find some
memory show summary
output.

Thank you for your help.

If you need further information please ask.

Best Regards
Thomas




****** ADDITIONAL INFORMATION ******

After about 15 min uptime:

{noformat}
*CLI> memory show summary
     3384 bytes in 1 allocations in file 'res_calendar.c'
    28472 bytes in 2 allocations in file 'app_followme.c'
     6240 bytes in 1 allocations in file 'tcptls.c'
      784 bytes in 2 allocations in file 'res_crypto.c'
      248 bytes in 1 allocations in file 'iax2-provision.c'
     3096 bytes in 1 allocations in file 'chan_local.c'
    18560 bytes in 1 allocations in file 'stdtime/localtime.c'
      240 bytes in 2 allocations in file 'cdr.c'
     3104 bytes in 1 allocations in file 'pbx_ael.c'
     3672 bytes in 1 allocations in file 'func_dialgroup.c'
   117624 bytes in 1 allocations in file 'ssl.c'
      280 bytes in 5 allocations in file 'devicestate.c'
        1 bytes in 1 allocations in file 'app_dial.c'
     9720 bytes in 2 allocations in file 'features.c'
     4632 bytes in 1 allocations in file 'cel.c'
       15 bytes in 3 allocations in file 'http.c'
      100 bytes in 5 allocations in file 'chan_misdn.c'
     3160 bytes in 5 allocations in file 'timing.c'
    12088 bytes in 1 allocations in file 'res_timing_pthread.c'
    22840 bytes in 6 allocations in file 'mpool/mpool.c'
    40423 bytes in 19 allocations in file 'taskprocessor.c'
    12248 bytes in 1 allocations in file 'pbx_realtime.c'
    12513 bytes in 6 allocations in file 'logger.c'
    28312 bytes in 11 allocations in file 'channel.c'
     6768 bytes in 6 allocations in file 'ccss.c'
     3928 bytes in 1 allocations in file 'app_confbridge.c'
    33376 bytes in 44 allocations in file 'hashtab.c'
   188905 bytes in 53 allocations in file 'data.c'
     3626 bytes in 4 allocations in file 'chan_oss.c'
     6072 bytes in 23 allocations in file 'file.c'
    27638 bytes in 9 allocations in file 'dnsmgr.c'
    19665 bytes in 26 allocations in file 'res_odbc.c'
    36011 bytes in 10 allocations in file 'app_minivm.c'
    38924 bytes in 28 allocations in file 'app_voicemail.c'
    24688 bytes in 13 allocations in file 'io.c'
     1578 bytes in 38 allocations in file 'cdr_mysql.c'
    23299 bytes in 13 allocations in file 'func_odbc.c'
   264312 bytes in 64 allocations in file 'chan_dahdi.c'
     3096 bytes in 1 allocations in file 'bridge_multiplexed.c'
     9608 bytes in 62 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/threadstorage.h'
    32160 bytes in 30 allocations in file 'sig_pri.c'
     3176 bytes in 43 allocations in file 'utils.c'
   127342 bytes in 106 allocations in file 'manager.c'
     1933 bytes (      1933 cache) in 4 allocations in file 'frame.c'
    17016 bytes in 72 allocations in file 'res_config_mysql.c'
    12694 bytes in 322 allocations in file 'asterisk.c'
     7861 bytes in 4 allocations in file 'res_musiconhold.c'
     9224 bytes in 3 allocations in file 'netsock.c'
    81745 bytes in 33 allocations in file 'app_queue.c'
    29952 bytes in 104 allocations in file 'acl.c'
   487184 bytes in 539 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/strings.h'
  1292238 bytes in 1199 allocations in file 'indications.c'
   569089 bytes in 542 allocations in file 'loader.c'
     3238 bytes in 89 allocations in file 'misdn_config.c'
   170980 bytes in 229 allocations in file 'config.c'
101698408 bytes in 32951 allocations in file 'chan_iax2.c'
    10866 bytes in 555 allocations in file 'cli.c'
    38505 bytes in 1479 allocations in file 'ael/pval.c'
    84594 bytes in 917 allocations in file 'xmldoc.c'
  4423967 bytes in 7493 allocations in file 'pbx.c'
   820320 bytes in 34180 allocations in file 'astobj2.c'
   180464 bytes in 1473 allocations in file 'sched.c'
  7202097 bytes in 3170 allocations in file 'chan_sip.c'
  1176501 bytes in 768 allocations in file 'event.c'
119504804 bytes allocated (1933 in caches) in 86780 allocations
{noformat}

After 15 hours:

{noformat}
voip01*CLI> core show uptime
System uptime: 15 hours, 37 minutes, 36 seconds
Last reload: 15 hours, 37 minutes, 36 seconds

voip01*CLI> memory show summary
     3384 bytes in 1 allocations in file 'res_calendar.c'
    28472 bytes in 2 allocations in file 'app_followme.c'
      784 bytes in 2 allocations in file 'res_crypto.c'
      248 bytes in 1 allocations in file 'iax2-provision.c'
     3096 bytes in 1 allocations in file 'chan_local.c'
    37120 bytes in 2 allocations in file 'stdtime/localtime.c'
      240 bytes in 2 allocations in file 'cdr.c'
     3104 bytes in 1 allocations in file 'pbx_ael.c'
     3672 bytes in 1 allocations in file 'func_dialgroup.c'
   117624 bytes in 1 allocations in file 'ssl.c'
      280 bytes in 5 allocations in file 'devicestate.c'
    12513 bytes in 6 allocations in file 'logger.c'
        1 bytes in 1 allocations in file 'app_dial.c'
     9720 bytes in 2 allocations in file 'features.c'
     7768 bytes in 1 allocations in file 'rtp_engine.c'
     4632 bytes in 1 allocations in file 'cel.c'
       15 bytes in 3 allocations in file 'http.c'
      100 bytes in 5 allocations in file 'chan_misdn.c'
     3160 bytes in 5 allocations in file 'timing.c'
     9544 bytes in 2 allocations in file 'res_rtp_asterisk.c'
    12088 bytes in 1 allocations in file 'res_timing_pthread.c'
    22840 bytes in 6 allocations in file 'mpool/mpool.c'
    40423 bytes in 19 allocations in file 'taskprocessor.c'
    12248 bytes in 1 allocations in file 'pbx_realtime.c'
    28312 bytes in 11 allocations in file 'channel.c'
     6768 bytes in 6 allocations in file 'ccss.c'
     3928 bytes in 1 allocations in file 'app_confbridge.c'
    33376 bytes in 44 allocations in file 'hashtab.c'
   188905 bytes in 53 allocations in file 'data.c'
     3626 bytes in 4 allocations in file 'chan_oss.c'
     6072 bytes in 23 allocations in file 'file.c'
    27638 bytes in 9 allocations in file 'dnsmgr.c'
    19665 bytes in 26 allocations in file 'res_odbc.c'
    36011 bytes in 10 allocations in file 'app_minivm.c'
    42796 bytes in 31 allocations in file 'app_voicemail.c'
    24688 bytes in 13 allocations in file 'io.c'
     1578 bytes in 38 allocations in file 'cdr_mysql.c'
    23299 bytes in 13 allocations in file 'func_odbc.c'
   264312 bytes in 64 allocations in file 'chan_dahdi.c'
     3096 bytes in 1 allocations in file 'bridge_multiplexed.c'
    32160 bytes in 30 allocations in file 'sig_pri.c'
     3098 bytes in 42 allocations in file 'utils.c'
   116759 bytes in 103 allocations in file 'manager.c'
    21152 bytes (      4904 cache) in 13 allocations in file 'frame.c'
    12670 bytes in 321 allocations in file 'asterisk.c'
     7861 bytes in 4 allocations in file 'res_musiconhold.c'
     9224 bytes in 3 allocations in file 'netsock.c'
    81745 bytes in 33 allocations in file 'app_queue.c'
    44058 bytes in 15 allocations in file 'app_meetme.c'
    29952 bytes in 104 allocations in file 'acl.c'
  1292238 bytes in 1199 allocations in file 'indications.c'
   569089 bytes in 542 allocations in file 'loader.c'
     3238 bytes in 89 allocations in file 'misdn_config.c'
    11232 bytes in 76 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/threadstorage.h'
  1502866 bytes in 978 allocations in file 'event.c'
   171172 bytes in 233 allocations in file 'config.c'
101698408 bytes in 32951 allocations in file 'chan_iax2.c'
    10866 bytes in 555 allocations in file 'cli.c'
    38505 bytes in 1479 allocations in file 'ael/pval.c'
    84594 bytes in 917 allocations in file 'xmldoc.c'
  4426876 bytes in 7543 allocations in file 'pbx.c'
   463112 bytes in 515 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/strings.h'
103034997 bytes in 38058 allocations in file 'chan_sip.c'
   117564 bytes in 1269 allocations in file 'res_config_mysql.c'
   180816 bytes in 1468 allocations in file 'sched.c'
  1032384 bytes in 43016 allocations in file 'astobj2.c'
216043782 bytes allocated (4904 in caches) in 131975 allocations
{noformat}
Comments:By: Dinesh Ramjuttun (din3sh) 2011-01-28 03:29:03.000-0600

I am also experiencing performance problems.

Is your server's load average also high (as compared to asterisk 1.4.x)??

For me on same hardware/server, load average on server(running Centos 5.5 x86_64) for 1min, 5mins, 15 mins are (using "top" command):

asterisk 1.4.22/zaptel 1.4.12.1 = load average: 0.16, 0.13, 0.11
asterisk 1.8.2.3/dahdi 2.4       = load average: 0.47, 0.39, 0.29


By: Thomas Z. (zircon) 2011-01-30 08:49:32.000-0600

{noformat}
voip01*CLI> core show uptime
System uptime: 2 days, 21 hours, 33 minutes, 53 seconds
Last reload: 2 days, 21 hours, 33 minutes, 53 seconds
voip01*CLI> memory show summary
     3384 bytes in 1 allocations in file 'res_calendar.c'
    28472 bytes in 2 allocations in file 'app_followme.c'
      784 bytes in 2 allocations in file 'res_crypto.c'
     6240 bytes in 1 allocations in file 'tcptls.c'
      248 bytes in 1 allocations in file 'iax2-provision.c'
     3096 bytes in 1 allocations in file 'chan_local.c'
    37120 bytes in 2 allocations in file 'stdtime/localtime.c'
      240 bytes in 2 allocations in file 'cdr.c'
     3104 bytes in 1 allocations in file 'pbx_ael.c'
     3672 bytes in 1 allocations in file 'func_dialgroup.c'
   117624 bytes in 1 allocations in file 'ssl.c'
    12513 bytes in 6 allocations in file 'logger.c'
      280 bytes in 5 allocations in file 'devicestate.c'
        1 bytes in 1 allocations in file 'app_dial.c'
     9720 bytes in 2 allocations in file 'features.c'
     4632 bytes in 1 allocations in file 'cel.c'
       15 bytes in 3 allocations in file 'http.c'
      100 bytes in 5 allocations in file 'chan_misdn.c'
     3160 bytes in 5 allocations in file 'timing.c'
    12088 bytes in 1 allocations in file 'res_timing_pthread.c'
    22840 bytes in 6 allocations in file 'mpool/mpool.c'
    40423 bytes in 19 allocations in file 'taskprocessor.c'
    12248 bytes in 1 allocations in file 'pbx_realtime.c'
    28312 bytes in 11 allocations in file 'channel.c'
     6768 bytes in 6 allocations in file 'ccss.c'
     3928 bytes in 1 allocations in file 'app_confbridge.c'
    36576 bytes in 44 allocations in file 'hashtab.c'
   188905 bytes in 53 allocations in file 'data.c'
     3626 bytes in 4 allocations in file 'chan_oss.c'
     6072 bytes in 23 allocations in file 'file.c'
    27638 bytes in 9 allocations in file 'dnsmgr.c'
    19665 bytes in 26 allocations in file 'res_odbc.c'
    36011 bytes in 10 allocations in file 'app_minivm.c'
    45879 bytes in 32 allocations in file 'app_voicemail.c'
    24688 bytes in 13 allocations in file 'io.c'
     1578 bytes in 38 allocations in file 'cdr_mysql.c'
    23299 bytes in 13 allocations in file 'func_odbc.c'
   264312 bytes in 64 allocations in file 'chan_dahdi.c'
     3096 bytes in 1 allocations in file 'bridge_multiplexed.c'
    32160 bytes in 30 allocations in file 'sig_pri.c'
     3176 bytes in 43 allocations in file 'utils.c'
   127689 bytes in 108 allocations in file 'manager.c'
     5252 bytes (      5252 cache) in 13 allocations in file 'frame.c'
    12694 bytes in 322 allocations in file 'asterisk.c'
     7861 bytes in 4 allocations in file 'res_musiconhold.c'
     9224 bytes in 3 allocations in file 'netsock.c'
    81745 bytes in 33 allocations in file 'app_queue.c'
    58744 bytes in 20 allocations in file 'app_meetme.c'
    30528 bytes in 106 allocations in file 'acl.c'
  1292238 bytes in 1199 allocations in file 'indications.c'
   569089 bytes in 542 allocations in file 'loader.c'
     3238 bytes in 89 allocations in file 'misdn_config.c'
    14752 bytes in 93 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/threadstorage.h'
   171172 bytes in 233 allocations in file 'config.c'
101698408 bytes in 32951 allocations in file 'chan_iax2.c'
    10866 bytes in 555 allocations in file 'cli.c'
    38505 bytes in 1479 allocations in file 'ael/pval.c'
    84594 bytes in 917 allocations in file 'xmldoc.c'
  4431434 bytes in 7644 allocations in file 'pbx.c'
   487286 bytes in 546 allocations in file '/usr/src/asterisk-1.8.2.3/include/asterisk/strings.h'
   467342 bytes in 5433 allocations in file 'res_config_mysql.c'
   186760 bytes in 1548 allocations in file 'sched.c'
  1805712 bytes in 75238 allocations in file 'astobj2.c'
  2469486 bytes in 1600 allocations in file 'event.c'
453378898 bytes in 165724 allocations in file 'chan_sip.c'
568521210 bytes allocated (5252 in caches) in 296891 allocations
voip01*CLI>
{noformat}

By: Leif Madsen (lmadsen) 2011-01-31 14:54:09.000-0600

OK I was talking to Tilghman and he requested the following:

MALLOC_DEBUG (Compiler Options, menuselect) and 'memory show summary' before and after the memory leak.
By: Thomas Z. (zircon) 2011-02-01 05:04:01.000-0600

Hi Imadsen,

I uploaded a logoutput after Astersik uptime of 1 day 20 hours.
We updated the Asterisk Version from 1.4 to 1.8.
After that we had the memory leak.
Therfor we can not provide the log before the memory leak.

If you need further information do not heasitate to contact me.

Regards
Thomas
By: John Covert (jcovert) 2011-02-11 00:35:04.000-0600

This may be a duplicate of issue ASTERISK-16710 where I uploaded the growth pattern we are seeing on our main production server.

I tried tonight to turn on malloc_debug, but that causes a crash in res_config_ldap, which is reported as issue ASTERISK-17386.

/john
By: Thomas Z. (zircon) 2011-03-21 02:52:22

Any news regarding this Bug?
We must restart every 1 - 2 days.
Regards
Thomas
By: Florent Chauveau (mav3rick) 2011-04-10 10:26:21

Have you noticed this line :

101698408 bytes in 32951 allocations in file 'chan_iax2.c'

I suppose you are not using IAX2.

I have the same memory leak with Asterisk 1.6.2.17, where I don't use IAX2 and it keeps leaking.

I believe I found a workaround adding "noload => chan_iax2.so" to modules.conf, although I can't be sure.
By: Thomas Z. (zircon) 2011-04-11 01:56:02

Hi mav3rick,

we are also using IAX2.
The big problem is SIP.
We must restart Asterisk every 2 Days. :-(

Thanks
Regards
Thomas
By: Leif Madsen (lmadsen) 2011-05-09 09:19:49

Can someone verify this is an issue with the latest 1.8 branch, and not directly on a release version? The reason I ask is the releases are quite a bit (maybe 6-8 weeks) behind the branch currently. Some security releases put the 1.8 tags behind a bit recently.
By: Thomas Z. (zircon) 2011-08-22 01:30:26.092-0500

Hello,
we updated a few days ago to *1.8.5.0* and the memory leak is not present any more.
So you can close this bug.
Regards
Thomas

By: Thomas Z. (zircon) 2011-09-07 07:27:57.125-0500

Hello,

cause of Bug ASTERISK-18420 we must go back to 1.8.3.3 and the memory leak is still present. :(

Regards
Thomas