[Home]

Summary:ASTERISK-12453: Memory leak in iax2-parser.c
Reporter:Octavio Ascanio (tavius)Labels:
Date Opened:2008-07-25 10:10:12Date Closed:2008-07-31 11:37:25
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Channels/chan_iax2
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 13160-debug.patch1.txt
( 1) 20080730__bug13160.diff.txt
Description:We have found compiling asterisk without LOW_MEMORY has a memory leak in iax_frame_new. It seems that the AST_LIST_TRAVERSE_SAFE_BEGIN macro is not traversing the linked list in good way.
We have traced adding some ast_log in the code that iax_frame_free add buffers in the frame cache linked list and after that iax_frame_new does not find this free buffer in the list and allocates a new buffer with ast_calloc_cache.
With the time and with our system, that receives too many calls, asterisk gets all the available memory in the system and produces an error.
If we compile with LOW_MEMORY, then this piece of code is not compiled and asterisk does not have memory leaks.
Comments:By: Octavio Ascanio (tavius) 2008-07-25 14:04:33

It seems that chan_iax creates several threads and there are threads that free buffers allocated by other threads so at the end there are threads allocating memory because they have no buffers in their linked list and another threads that have many buffers in his list and are adding more buffers to the list.
Russell Bryant has provided us with a patch that limits the max items on the linked list to 20 but this is a workaround to limit the memory leak, not  a solution.
By: Russell Bryant (russell) 2008-07-29 07:23:44

For anyone looking at debugging this, the first thing that I wanted to do was check to see if there are any threads in chan_iax2 that only free frames, or free a lot more than they allocate.  In that case, the iax_frame_free function could take a parameter to hint whether the frame should be cached or not.  See main/frame.c for similar handling.  If that won't work for some reason, then the patch I posted to the -dev list to limit the cache size is probably the next best thing.
By: Russell Bryant (russell) 2008-07-29 07:24:43

http://lists.digium.com/pipermail/asterisk-dev/2008-July/034080.html
By: Russell Bryant (russell) 2008-07-30 15:12:03

I have uploaded a patch that will add some additional debug information.  Please apply this and let me know what the text is of the ERROR log messages that show up.  It will print out the name of the thread that has a full frame cache.
By: Tilghman Lesher (tilghman) 2008-07-30 16:35:42

Heh, I was working on one, as well.
By: Octavio Ascanio (tavius) 2008-07-31 07:41:22

ops, first call, three seconds and this:

[2008-07-31 13:47:37] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:37] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:37] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:37] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:37] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:37] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:38] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:38] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:39] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:39] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:39] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:39] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
[2008-07-31 13:47:39] ERROR[27861]: iax2-parser.c:1034 iax_frame_free: Frame cache full for thread 3066825616
[2008-07-31 13:47:39] ERROR[27861]: utils.c:861 ast_log_thread_info: Thread 'iax2_process_thread  started at [ 9130] chan_iax2.c start_network_thread()' ID '3066825616'
By: Russell Bryant (russell) 2008-07-31 11:32:03

Thanks for the quick response.
By: Digium Subversion (svnbot) 2008-07-31 11:37:22

Repository: asterisk
Revision: 134814

U   branches/1.4/channels/iax2-parser.c

------------------------------------------------------------------------
r134814 | russell | 2008-07-31 11:37:22 -0500 (Thu, 31 Jul 2008) | 7 lines

In case we have some processing threads that free more frames than they allocate,
do not let the frame cache grow forever.

(closes issue ASTERISK-12453)
Reported by: tavius
Tested by: tavius, russell

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=134814