[Home]

Summary:ASTERISK-13743: MOH Realtime crash
Reporter:Sebastian Gutierrez (sum)Labels:
Date Opened:2009-03-13 09:43:45Date Closed:2009-04-09 14:15:51
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Resources/res_musiconhold
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20090316__bug14661.diff.txt
( 1) dump.txt
( 2) malloc_debug162.txt
( 3) memoriclasses.txt
( 4) valgrind.txt
( 5) valgrind162.txt
( 6) valgrind2.txt
( 7) valgrind3.txt
( 8) valgrind4.txt
( 9) valgrindcore.txt
Description:Situation:

Queue realtime, musiconhold realtime class = prueba
I have default class on musiconhold.conf
When a call is made to the Queue checks that is not on memory so goes to the db, I had 2 situations:

1) If digit was #. Asterisk crash (check dump.txt)
2) With no digit, when I check the classes (check memoryclasses.txt)

seems bad memory managment.
Comments:By: Tilghman Lesher (tilghman) 2009-03-13 11:44:50

Please read and follow the instructions in doc/valgrind.txt.
By: Sebastian Gutierrez (sum) 2009-03-13 12:02:02

whe I try to

valgrind --log-file-exactly=valgrind.txt asterisk -vvvvcg 2>malloc_debug.txt

is Killed and I cant reproduce the problem, I have a valgrind core.
my valgrind version is valgrind-3.2.1
By: Sebastian Gutierrez (sum) 2009-03-13 12:07:04

I attach the bt of the valgrind core : valgrindcore.txt
By: Tilghman Lesher (tilghman) 2009-03-13 12:41:35

Please install SVN branch 1.6.0 of Asterisk-addons.  I believe this has already been fixed, although a new release has not yet been made.
By: Sebastian Gutierrez (sum) 2009-03-13 14:37:23

1) Asterisk 1.6.0.6
2) Asterisk addons 1.6.0 svn branch

asterisk crashes, if I use asterisk -vvvvvvv to se where it does:

 == Parsing '/etc/asterisk/cdr_mysql.conf':   == Found
WARNING: Realloc of unalloced memory at 0x9d22e88, in ast_str_make_space of /usr/src/asterisk-1.6.0.6/include/asterisk/strings.h, line 433
Segmentation fault

Any idea?


By: Sebastian Gutierrez (sum) 2009-03-13 14:52:22

1) Asterisk 1.6.0.6
2) Asterisk addons 1.6.0.1

executing with valgrind (valgrind3.txt), i reproduce the crash making some calls to the queue and doing moh show classes that shows me garbage.
By: Sebastian Gutierrez (sum) 2009-03-16 17:08:59

any update on this issue?
By: Tilghman Lesher (tilghman) 2009-03-16 18:14:48

You need to run both the SVN versions of Asterisk-1.6.0 and Asterisk-addons-1.6.0.
By: Sebastian Gutierrez (sum) 2009-03-16 18:45:43

same problem with 1.6.0 SVN Asterisk and Asterisk addons, do you want more information? valgrind? let me know.
By: Tilghman Lesher (tilghman) 2009-03-16 19:28:28

Try this patch.
By: Sebastian Gutierrez (sum) 2009-03-16 19:52:02

is this patch for SVN 1.6.0? cause it doesn't apply to it.


UPDATE: the part of res_musiconhold, seems to apply ok, but chan_sip is different, I tried with just the res_music patch and asterisk crashes the first time it tryes to reach RT moh.


By: Sebastian Gutierrez (sum) 2009-03-19 09:01:12

tell me if you need any test on a different version.
I can test your patches.
By: Sebastian Gutierrez (sum) 2009-03-20 21:12:22

the issue still persist on last released versions 1.6.2 beta 1 and the last addons,
By: Sebastian Gutierrez (sum) 2009-03-22 15:41:50

patch applyed cleanly to 1.6.2 beta 1 (not sure if was supposed to be for this release too but anyway I tried cause it seems ok for it)

The beheavior still remains I attach valgrind of the dump with DON'T OPTIMIZE (remember malloc debug is not working with 1.6.2-beta1 and realtime agents).

i can see on the cli:
   -- Started music on hold, class 'prueba', on SIP/1001-07058430
[Mar 21 09:44:59] ERROR[4729]: astobj2.c:110 INTERNAL_OBJ: user_data is NULL


valgrind4.txt uploaded
By: Sebastian Gutierrez (sum) 2009-03-29 01:07:01

the problems seems to be with cachertclasses=yes, if I comment it out evertything works ok, although I have in the cli this:

[Mar 28 08:09:08] ERROR[25247]: astobj2.c:110 INTERNAL_OBJ: user_data is NULL
[Mar 28 08:09:08] DEBUG[25247]: res_musiconhold.c:747 get_mohbyname: Music on Hold class 'prueba' not found in memory
   -- Started music on hold, class 'prueba', on SIP/1001-09a9d0f0
[Mar 28 08:09:08] ERROR[25247]: astobj2.c:110 INTERNAL_OBJ: user_data is NULL

But there is no crash, could this be a problem of astobj2 to cache the moh class to memory? if I check moh show clases after a first try with cache enable I can see that the cached moh class is filled with garbage data, and not the real data of the class, also after is cached the second time never find it in memory (due to what I can see at moh show clasess) so it goes feeling the moh clasess list until it crashes.
By: Sebastian Gutierrez (sum) 2009-04-01 17:18:46

Tested with addons 1.6.2 beta1 and same issue, I don't think has nothing to do with addons but just to inform the tests made. please if you need further info let me know
By: Sebastian Gutierrez (sum) 2009-04-02 20:25:31

tested with addons 1.6.2 and Asterisk SVN-branch-1.6.2-r185947M
Uploaded: valgrind162 and mallocdebug162

should I try the patch uploaded with this version?


By: Digium Subversion (svnbot) 2009-04-09 12:30:41

Repository: asterisk
Revision: 187421

U   trunk/res/res_musiconhold.c

------------------------------------------------------------------------
r187421 | mmichelson | 2009-04-09 12:30:41 -0500 (Thu, 09 Apr 2009) | 21 lines

Fix a crash in res_musiconhold when using cached realtime moh.

The moh_register function links an mohclass and then immediately
unrefs the class since the container now has a reference. The problem
with using realtime music on hold is that the class is allocated,
registered, and started in one fell swoop. The refcounting logic
resulted in the count being off by one. The same problem did not
happen when using a static config because the allocation and registration
of an mohclass is a separate operation from starting moh. This also did
not affect non-cached realtime moh because the classes are not registered
at all.

I also have modified res_musiconhold to use the _t_ variants of the ao2_
functions so that more info can be gleaned when attempting to trace the
refcounts. I found this to be incredibly helpful for debugging this issue
and there's no good reason to remove it.

(closes issue ASTERISK-13743)
Reported by: sum


------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=187421
By: Digium Subversion (svnbot) 2009-04-09 12:37:51

Repository: asterisk
Revision: 187425

_U  branches/1.6.0/
U   branches/1.6.0/res/res_musiconhold.c

------------------------------------------------------------------------
r187425 | mmichelson | 2009-04-09 12:37:51 -0500 (Thu, 09 Apr 2009) | 31 lines

Merged revisions 187421,187424 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r187421 | mmichelson | 2009-04-09 12:30:39 -0500 (Thu, 09 Apr 2009) | 21 lines
 
 Fix a crash in res_musiconhold when using cached realtime moh.
 
 The moh_register function links an mohclass and then immediately
 unrefs the class since the container now has a reference. The problem
 with using realtime music on hold is that the class is allocated,
 registered, and started in one fell swoop. The refcounting logic
 resulted in the count being off by one. The same problem did not
 happen when using a static config because the allocation and registration
 of an mohclass is a separate operation from starting moh. This also did
 not affect non-cached realtime moh because the classes are not registered
 at all.
 
 I also have modified res_musiconhold to use the _t_ variants of the ao2_
 functions so that more info can be gleaned when attempting to trace the
 refcounts. I found this to be incredibly helpful for debugging this issue
 and there's no good reason to remove it.
 
 (closes issue ASTERISK-13743)
 Reported by: sum
........
 r187424 | mmichelson | 2009-04-09 12:34:39 -0500 (Thu, 09 Apr 2009) | 3 lines
 
 Use safe macro practices even though they really aren't necessary.
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=187425
By: Digium Subversion (svnbot) 2009-04-09 12:43:21

Repository: asterisk
Revision: 187427

_U  branches/1.6.1/
U   branches/1.6.1/res/res_musiconhold.c

------------------------------------------------------------------------
r187427 | mmichelson | 2009-04-09 12:43:21 -0500 (Thu, 09 Apr 2009) | 31 lines

Merged revisions 187421,187424 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r187421 | mmichelson | 2009-04-09 12:30:39 -0500 (Thu, 09 Apr 2009) | 21 lines
 
 Fix a crash in res_musiconhold when using cached realtime moh.
 
 The moh_register function links an mohclass and then immediately
 unrefs the class since the container now has a reference. The problem
 with using realtime music on hold is that the class is allocated,
 registered, and started in one fell swoop. The refcounting logic
 resulted in the count being off by one. The same problem did not
 happen when using a static config because the allocation and registration
 of an mohclass is a separate operation from starting moh. This also did
 not affect non-cached realtime moh because the classes are not registered
 at all.
 
 I also have modified res_musiconhold to use the _t_ variants of the ao2_
 functions so that more info can be gleaned when attempting to trace the
 refcounts. I found this to be incredibly helpful for debugging this issue
 and there's no good reason to remove it.
 
 (closes issue ASTERISK-13743)
 Reported by: sum
........
 r187424 | mmichelson | 2009-04-09 12:34:39 -0500 (Thu, 09 Apr 2009) | 3 lines
 
 Use safe macro practices even though they really aren't necessary.
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=187427
By: Digium Subversion (svnbot) 2009-04-09 14:15:50

Repository: asterisk
Revision: 187496

_U  branches/1.6.2/
U   branches/1.6.2/res/res_musiconhold.c

------------------------------------------------------------------------
r187496 | mmichelson | 2009-04-09 14:15:50 -0500 (Thu, 09 Apr 2009) | 31 lines

Merged revisions 187421,187424 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r187421 | mmichelson | 2009-04-09 12:30:39 -0500 (Thu, 09 Apr 2009) | 21 lines
 
 Fix a crash in res_musiconhold when using cached realtime moh.
 
 The moh_register function links an mohclass and then immediately
 unrefs the class since the container now has a reference. The problem
 with using realtime music on hold is that the class is allocated,
 registered, and started in one fell swoop. The refcounting logic
 resulted in the count being off by one. The same problem did not
 happen when using a static config because the allocation and registration
 of an mohclass is a separate operation from starting moh. This also did
 not affect non-cached realtime moh because the classes are not registered
 at all.
 
 I also have modified res_musiconhold to use the _t_ variants of the ao2_
 functions so that more info can be gleaned when attempting to trace the
 refcounts. I found this to be incredibly helpful for debugging this issue
 and there's no good reason to remove it.
 
 (closes issue ASTERISK-13743)
 Reported by: sum
........
 r187424 | mmichelson | 2009-04-09 12:34:39 -0500 (Thu, 09 Apr 2009) | 3 lines
 
 Use safe macro practices even though they really aren't necessary.
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=187496