Summary:ASTERISK-26078: core: Memory leak in logging
Reporter:Etienne Lessard (hexanol)Labels:
Date Opened:2016-05-31 14:58:16Date Closed:2016-06-01 16:13:02
Versions:13.8.0 13.9.1 Frequency of
is duplicated byASTERISK-26150 Memory leakage in stringfields.c and logger.c
Environment:Debian 8 amd64Attachments:( 0) jira_asterisk_26078_v13.patch
( 1) mmlog
( 2) mmsummary1
( 3) mmsummary2
( 4) valgrind.txt
Description:There seems to be a new memory leak since Asterisk 13.8 or 13.9.

Running a simple call center scenario with sipp (a new call into a queue every few seconds, that is answered by a queue member, etc), I'm seeing asterisk memory usage grow noticeably (a little bit less than 100 MB/hour).

Looking at a diff from two "memory show summary" commands taken 30 minutes apart, it looks like the leak comes from logger.c / stringfields.c. I've attached the output of two "memory show summary".

I've also attached the mmlog file, but I doubt this will be useful since it's mostly empty (guess I've done something wrong).

I'm already aware of ASTERISK-25262, but this is not it: I've explicitly disabled the CEL in my current tests. I've also seen ASTERISK-25905, but this is not it (I've also try manually running malloc_trim, it gave nothing). Maybe it's similar to ASTERISK-26020, but there's not much info in that ticket.

Thank you
Comments:By: Asterisk Team (asteriskteam) 2016-05-31 14:58:17.578-0500

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

By: Etienne Lessard (hexanol) 2016-05-31 15:03:00.368-0500

I also had a valgrind output from a previous run, so I'll also attach it even though I'm not sure if it helps (that said, we do see something related to logger.c and stringfields.c, e.g.):

==11069== 1,409,504 (626,672 direct, 782,832 indirect) bytes in 39,167 blocks are definitely lost in loss record 9,148 of 9,151
==11069==    at 0x402B0D5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==11069==    by 0x820ECF6: _ast_calloc (in /usr/sbin/asterisk)
==11069==    by 0x81F9203: calloc_wrapper (in /usr/sbin/asterisk)
==11069==    by 0x81F9CB2: __ast_calloc_with_stringfields (in /usr/sbin/asterisk)
==11069==    by 0x816328A: ast_log_full (in /usr/sbin/asterisk)
==11069==    by 0x8163ED8: ast_log_callid (in /usr/sbin/asterisk)
==11069==    by 0x8164235: __ast_verbose_ap (in /usr/sbin/asterisk)
==11069==    by 0x816429A: __ast_verbose (in /usr/sbin/asterisk)
==11069==    by 0x8191D33: pbx_extension_helper (in /usr/sbin/asterisk)
==11069==    by 0x8194DF6: ast_spawn_extension (in /usr/sbin/asterisk)
==11069==    by 0x8195A9F: __ast_pbx_run (in /usr/sbin/asterisk)
==11069==    by 0x81971CA: pbx_thread (in /usr/sbin/asterisk)

My tests were done on Asterisk 13.9.1.

By: Etienne Lessard (hexanol) 2016-05-31 17:18:10.236-0500

It seems like the logmsg_free function in logger.c is missing a call to "ast_string_field_free_memory".

I'm guessing this memory leak was a side effect of [commit f6f4cf459f43f072604927209b39646f84aaa2e2|https://github.com/asterisk/asterisk/commit/f6f4cf459f43f072604927209b39646f84aaa2e2]. I don't know if other parts of the source code are affected by a similar issue.

By: Richard Mudgett (rmudgett) 2016-05-31 18:22:42.723-0500

The following
are also missing the ast_string_field_free_memory() call.

By: Richard Mudgett (rmudgett) 2016-06-01 13:36:36.351-0500

[^jira_asterisk_26078_v13.patch] - This patch adds the missing ast_string_field_free_memory() call to logger.c, cel_custom.c, cdr_syslog.c, and cdr_custom.c.

Please report if this takes care of the memory leak.

By: Etienne Lessard (hexanol) 2016-06-01 13:52:47.070-0500

Please report if this takes care of the memory leak.

The patch in logger.c does take care of the memory leak I was observing.