[Home]

Summary:ASTERISK-15102: [patch] asterisk keeps starting new processes for streaming audio MOH
Reporter:Dave Cabot (dcabot)Labels:
Date Opened:2009-11-07 20:51:31.000-0600Date Closed:2009-12-02 18:40:18.000-0600
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Resources/res_musiconhold
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20091123__issue16207.diff.txt
Description:Using AsteriskNOW 1.5.0, FreePBX 2.6.0.0, Asterisk 1.6.0.17.
Added streaming category to MOH.

[Jazz1]
mode=custom
application=/usr/bin/mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010

Asterisk starts mpg123 in sets of twos.  Stopped at 22 instances.

[root@FreePBX-1 asterisk]# ps ax|grep mpg123
3469 ?        S      1:14 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3606 ?        S      1:10 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3607 ?        S      1:06 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3631 ?        S      1:06 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3633 ?        S      1:18 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3655 ?        S      1:05 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
3656 ?        S      1:05 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4067 ?        R      0:23 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4069 ?        R      0:29 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4106 ?        S      0:21 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4108 ?        S      0:21 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4140 ?        S      0:24 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4141 ?        S      0:33 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4171 ?        S      0:20 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4173 ?        S      0:23 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4195 ?        S      0:19 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4196 ?        R      0:19 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4237 ?        S      0:17 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4238 ?        S      0:17 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4338 ?        S      0:09 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4339 ?        S      0:13 /usr/bin mpg123 -q -s --mono -r 8000 -f 8192 -b 0 http://scfire-ntc-aa08.stream.aol.com:80/stream/1010
4513 pts/1    R+     0:00 grep mpg123

No calls in progress.
Comments:By: mustardman (mustardman) 2009-11-08 16:02:43.000-0600

I have observed this behaviour as well.  I was running 1.6.1.8 with 1.6.1.1 addons.  MPG123 version was 1.9.1.

My streaming source was different from what the OP is using.


By: Leif Madsen (lmadsen) 2009-11-09 08:44:05.000-0600

And what is the issue here? I think this is supposed to happen... it is multiple threads running for the same application.
By: mustardman (mustardman) 2009-11-09 13:36:28.000-0600

I'm no expert but 22 10meg processes running the SAME application doesn't sound like by design behaviour to me.  This is with NO calls in progress.

In my case, it does NOT load this way.  It starts with 1 application and gradually increases over about an hour.  Each time a call is put on hold a new process of the same application starts.  When the call ends the application does not go away.  Another call on hold and another process starts and so on and so forth.  I think even if NO calls are put on hold the processes still keep multiplying.  Just more slowly.


By: Dave Cabot (dcabot) 2009-11-09 19:50:02.000-0600

With no calls on hold, I think it might not be the way it was designed to have 22 instances of a streaming category running.  What more info do you need to confirm this as a bug?
By: David Woolley (davidw) 2009-11-10 05:51:49.000-0600

I think the bug you are reporting is a failure to terminate redundant MOH processes, rather than one of starting excess ones.

There was a recent issue about a problem with streamed MOH suddenly jumping which might be consistent with a file descriptor being cloned and left open in one thread.  It's not in the res_musiconhold category, so I couldn't find it quickly.

Incidentally, a lot of the 10MB is likely to be shared readonly, or unused copy on write pages.
By: mustardman (mustardman) 2009-11-13 19:51:23.000-0600

I don't know if each process is 10meg.  It is whatever size the mpg123 process is.  Just used 10Meg as an example as I don't remember the size and since finding that problem I have abandoned any hope of using MOH streaming for now.


By: Leif Madsen (lmadsen) 2009-11-19 09:44:46.000-0600

Based on what is being said in issue ASTERISK-15170 could this be happening due to Asterisk being reloaded? I'm presuming if you're using FreePBX or some other GUI that it is reloading Asterisk when configuration changes are happening, which could be the root cause of this.
By: Leif Madsen (lmadsen) 2009-11-23 20:16:40.000-0600

For those who may not have noticed, a patch has been attached to this issue, and is now ready for testing.
By: Andrew Parisio (parisioa) 2009-11-24 15:42:55.000-0600

this patch does not fix 0016279
By: Tilghman Lesher (tilghman) 2009-11-30 11:05:41.000-0600

I wasn't trying to fix ASTERISK-15170, so whether this patch fixes that or not is irrelevant.  This patch still needs testing by those experiencing THIS issue.
By: Digium Subversion (svnbot) 2009-12-02 18:16:33.000-0600

Repository: asterisk
Revision: 232660

U   trunk/include/asterisk/astobj2.h
U   trunk/res/res_musiconhold.c

------------------------------------------------------------------------
r232660 | tilghman | 2009-12-02 18:16:31 -0600 (Wed, 02 Dec 2009) | 19 lines

Fix multiple issues with musiconhold, which led to classes not getting destroyed properly.
* Classes are now tracked past removal from the core container, and module
  removal is actively prevented until all references are freed.
* A hanging reference stored in the channel has been removed.  This could have
  caused a mismatch and the music state not properly cleared, if two or more
  reloads occurred between MOH being stopped and MOH being restarted.
* In certain circumstances, duplicate classes were possible.
* A race existed at reload time between a process being killed and the thread
  responsible for reading from the related pipe respawning that process.
* Several reference counts have also been corrected.  At least one could have
  caused deleted classes to stick around forever, consuming resources.  This
  originally manifested as MOH external processes that were not killed at
  reload time.
(closes issue ASTERISK-15170, closes issue ASTERISK-15102)
Reported by: parisioa, dcabot
Patches:
      20091202__issue16279__2.diff.txt uploaded by tilghman (license 14)
Tested by: parisioa, tilghman

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=232660
By: Digium Subversion (svnbot) 2009-12-02 18:25:40.000-0600

Repository: asterisk
Revision: 232666

_U  branches/1.6.1/
U   branches/1.6.1/res/res_musiconhold.c

------------------------------------------------------------------------
r232666 | tilghman | 2009-12-02 18:25:38 -0600 (Wed, 02 Dec 2009) | 30 lines

Recorded merge of revisions 232660-232661 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r232660 | tilghman | 2009-12-02 18:08:55 -0600 (Wed, 02 Dec 2009) | 19 lines
 
 Fix multiple issues with musiconhold, which led to classes not getting destroyed properly.
  * Classes are now tracked past removal from the core container, and module
    removal is actively prevented until all references are freed.
  * A hanging reference stored in the channel has been removed.  This could have
    caused a mismatch and the music state not properly cleared, if two or more
    reloads occurred between MOH being stopped and MOH being restarted.
  * In certain circumstances, duplicate classes were possible.
  * A race existed at reload time between a process being killed and the thread
    responsible for reading from the related pipe respawning that process.
  * Several reference counts have also been corrected.  At least one could have
    caused deleted classes to stick around forever, consuming resources.  This
    originally manifested as MOH external processes that were not killed at
    reload time.
 (closes issue ASTERISK-15170, closes issue ASTERISK-15102)
  Reported by: parisioa, dcabot
  Patches:
        20091202__issue16279__2.diff.txt uploaded by tilghman (license 14)
  Tested by: parisioa, tilghman
........
 r232661 | tilghman | 2009-12-02 18:09:36 -0600 (Wed, 02 Dec 2009) | 2 lines
 
 Remove debugging line
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=232666
By: Digium Subversion (svnbot) 2009-12-02 18:27:31.000-0600

Repository: asterisk
Revision: 232675

_U  branches/1.6.2/
U   branches/1.6.2/res/res_musiconhold.c

------------------------------------------------------------------------
r232675 | tilghman | 2009-12-02 18:27:30 -0600 (Wed, 02 Dec 2009) | 30 lines

Recorded merge of revisions 232660-232661 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r232660 | tilghman | 2009-12-02 18:08:55 -0600 (Wed, 02 Dec 2009) | 19 lines
 
 Fix multiple issues with musiconhold, which led to classes not getting destroyed properly.
  * Classes are now tracked past removal from the core container, and module
    removal is actively prevented until all references are freed.
  * A hanging reference stored in the channel has been removed.  This could have
    caused a mismatch and the music state not properly cleared, if two or more
    reloads occurred between MOH being stopped and MOH being restarted.
  * In certain circumstances, duplicate classes were possible.
  * A race existed at reload time between a process being killed and the thread
    responsible for reading from the related pipe respawning that process.
  * Several reference counts have also been corrected.  At least one could have
    caused deleted classes to stick around forever, consuming resources.  This
    originally manifested as MOH external processes that were not killed at
    reload time.
 (closes issue ASTERISK-15170, closes issue ASTERISK-15102)
  Reported by: parisioa, dcabot
  Patches:
        20091202__issue16279__2.diff.txt uploaded by tilghman (license 14)
  Tested by: parisioa, tilghman
........
 r232661 | tilghman | 2009-12-02 18:09:36 -0600 (Wed, 02 Dec 2009) | 2 lines
 
 Remove debugging line
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=232675
By: Digium Subversion (svnbot) 2009-12-02 18:40:15.000-0600

Repository: asterisk
Revision: 232699

_U  branches/1.6.0/
U   branches/1.6.0/res/res_musiconhold.c

------------------------------------------------------------------------
r232699 | tilghman | 2009-12-02 18:40:14 -0600 (Wed, 02 Dec 2009) | 30 lines

Merged revisions 232660-232661 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

........
 r232660 | tilghman | 2009-12-02 18:08:55 -0600 (Wed, 02 Dec 2009) | 19 lines
 
 Fix multiple issues with musiconhold, which led to classes not getting destroyed properly.
  * Classes are now tracked past removal from the core container, and module
    removal is actively prevented until all references are freed.
  * A hanging reference stored in the channel has been removed.  This could have
    caused a mismatch and the music state not properly cleared, if two or more
    reloads occurred between MOH being stopped and MOH being restarted.
  * In certain circumstances, duplicate classes were possible.
  * A race existed at reload time between a process being killed and the thread
    responsible for reading from the related pipe respawning that process.
  * Several reference counts have also been corrected.  At least one could have
    caused deleted classes to stick around forever, consuming resources.  This
    originally manifested as MOH external processes that were not killed at
    reload time.
 (closes issue ASTERISK-15170, closes issue ASTERISK-15102)
  Reported by: parisioa, dcabot
  Patches:
        20091202__issue16279__2.diff.txt uploaded by tilghman (license 14)
  Tested by: parisioa, tilghman
........
 r232661 | tilghman | 2009-12-02 18:09:36 -0600 (Wed, 02 Dec 2009) | 2 lines
 
 Remove debugging line
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=232699