[Home]

Summary:ASTERISK-17076: Asterisk 1.8.0 locks up under OpenSUSE 10.3
Reporter:Craig Arno (craigarno)Labels:
Date Opened:2010-12-07 10:54:18.000-0600Date Closed:2011-06-07 14:04:41
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Resources/res_musiconhold
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) backtrace2.txt
( 1) backtrace-threads.txt
( 2) core-show-locks.txt
Description:Asterisk 1.8.0 locks up in less than 24 hours <twice> on my lightly used installation.

Asterisk 1.8.0-rc2 is stable on the same system.  So changes between RC2 and Release are likely contributing problems for my installation.

No Add-Ons, just stock 1.8.0 distribution.

In locked-up state:
- CLI (asterisk -vvvr) seems to still be alive in locked state.
- [core stop now] CLI command has no effect.  Asterisk process continues to stay resident.
- asterisk process does not respond to "kill <processID>"
- kill -9 is needed to remove asterisk prior to restarting.


My system:
 OpenSUSE 10.3 x64/AMD
 Athlon64 X2 5200+/8GB RAM/1.5TB disk
 Is stable (hardware and software)

I can provide OpenSUSE 10.3 ISO if needed (hard to find on OpenSUSE site).

I rolled back to RC2 since I can't afford to have my phone system continue to randomly crash every day.  Hopefully a future release will address this issue.

****** ADDITIONAL INFORMATION ******

I see these messages on my console which may or may not be relevant;

[Dec  7 08:04:08] WARNING[6487]: res_musiconhold.c:554 spawn_mp3: Found no files in '/var/lib/asterisk/moh'
[Dec  7 08:04:08] WARNING[6487]: res_musiconhold.c:627 monmp3thread: Unable to spawn mp3player
[Dec  7 08:12:28] WARNING[6487]: res_musiconhold.c:554 spawn_mp3: Found no files in '/var/lib/asterisk/moh'
[Dec  7 08:12:28] WARNING[6487]: res_musiconhold.c:627 monmp3thread: Unable to spawn mp3player
[Dec  7 08:20:48] WARNING[6487]: res_musiconhold.c:554 spawn_mp3: Found no files in '/var/lib/asterisk/moh'
[Dec  7 08:20:48] WARNING[6487]: res_musiconhold.c:627 monmp3thread: Unable to spawn mp3player


/var/lib/asterisk/moh does exist and contains files, so I'm not sure why this message keeps popping up.
Comments:By: Craig Arno (craigarno) 2010-12-07 11:18:39.000-0600

I just had 1.8.0-rc2 lock up.  So this may be a configuration issue on my end.

I've been working to get MusicOnHold(default) working, and it was, or so I thought.

My [musiconhold.conf] contains these active lines:
[default]
directory=/var/lib/asterisk/mohdefault
mode=files
random=yes

[mp3]
mode=custom
directory=/var/lib/asterisk/mohmp3
application=/usr/local/bin/mpg123 -q -r 8000 -f 8192 -b 2048 --mono -s

[stream]
mode=custom
directory=/var/lib/asterisk/moh
application=/usr/local/bin/mpg123 -q -s --mono -r 8000 -f 8192 -b 1024 http://93.182.176.123

And something in this configuration appears to be crashing Asterisk.  My [extensions.conf] addresses this configuration with:
exten => 27,1,Answer
exten => 27,2,MusicOnHold(default)
exten => 27,3,Hangup

I'd probably reduce this bug to "Minor" status unless someone thinks MusicOnHold() shouldn't be able to crash the system.

I restored my [musiconhold.conf] configuration to the "Sample" version which seems to not crash the installation.
By: Leif Madsen (lmadsen) 2010-12-07 13:53:13.000-0600

To move this forward I would probably provide 'core show locks' (which will be enabled if you enable the DEBUG_THREADS flag in menuselect within the Compiler Flags menu and reinstall Asterisk).

Also, you can provide backtrace information as well.

More info here:  https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace
By: Craig Arno (craigarno) 2010-12-08 01:58:08.000-0600

Okay, this may simplify debugging...

Before placing a call, I checked locks and found two processes deadlocked.  My guess is others will join deadlock shortly;

Documentation is in the uploaded "backtrace2.txt" file...


By: Craig Arno (craigarno) 2010-12-08 09:41:45.000-0600

BTW, there are two bugs pointed to with this data;

1 - A deadlock between threads

2 - The MOH Destructor hangs if the MOH process isn't found.

#2 is the reason I had to exit asterisk with a <CNTR-C>.  Asterisk was hung on exit by the MOH function.

My hope is both of these will be addressed with this one bug submittal.
By: Craig Arno (craigarno) 2010-12-08 22:29:20.000-0600

I updated my post in the Asterisk Forums to show how to get MP3 working, even with these bugs (which I still consider important if not serious).
http://forums.asterisk.org/viewtopic.php?f=1&t=76388&p=151455#p151455

My post covers lacking documentation (couldn't find what I needed in doc/ distribution directory), and how to partially work around this bug (i.e. find the one configuration that does work).  I say partially, because even with everything I documented, the system crashed again.

There are a lot of little tidbits of knowledge needed that aren't generally known by the outside world.  I shared what I learned and needed that took me almost 2 days to figure out.  I hope it helps someone else.  Meanwhile we still have a show-stopper bug.


By: Craig Arno (craigarno) 2010-12-08 22:45:15.000-0600

Even with a now working configuration, the 1.8.0 installation locks up with MOH.

Asterisk was so dead, CLI wasn't responding and <CTRL-C> wouldn't work at the console.

I was able to get backtrace-threads and core-show-locks from the hung Asterisk before kill -9 was used to remove the process.  I tried just "kill 3800" and "kill -HUP 3800" prior to "kill -9 3800".  To kill -HUP I received the message "Received HUP signal -- Reloading configs" alone, all by itself, on the console.

Files  backtrace-threads.txt  core-show-locks.txt are uploaded here to support this note.
By: Leif Madsen (lmadsen) 2011-01-18 10:16:58.000-0600

Is this still an issue on the latest 1.8 code?
By: Russell Bryant (russell) 2011-01-19 13:55:25.000-0600

I see that you are using res_timing_pthread.  Can you please try installing res_timing_timerfd (if supported on your system) or installing DAHDI and res_timing_dahdi?