[Home]

Summary:ASTERISK-18134: Reload hangs when using func_odbc for hints.
Reporter:Steven Wheeler (swheeler)Labels:
Date Opened:2011-07-13 09:10:33Date Closed:2011-10-26 18:01:35
Priority:MajorRegression?
Status:Closed/CompleteComponents:Core/PBX Functions/func_odbc
Versions:1.6.2.19 Frequency of
Occurrence		Constant
Related
Issues:
Environment:CentOSAttachments:( 0) debug.20110708.txt
Description:When we setup many hints using func_odbc and pattern matching we are unable to execute a reload, dialplan reload, module reload, etc.  If we try to do another reload we get an error saying 'The previous reload command didn't finish yet'. This has been causing quite a headache because the only way to load new configuration changes is to restart the asterisk process which can only be done in the middle of the night when the call volume drops to zero.  If we disable the func_odbc hints then we have no issues reloading.

Our BLF context:
{noformat}
[blf]
exten => _[A-Za-z0-9*#].,hint,${RAVON_BLF_PHONE_LIST(${EXTEN})}
{noformat}

The corresponding func_odbc configuration:
{noformat}
[BLF_PHONE_LIST]
prefix=RAVON
dsn=ravon_rt
readsql=SELECT phone_list FROM Blf_Phonelists WHERE CONCAT(company_id,'-',username)='${SQL_ESC(${ARG1})}'
{noformat}

Output of 'core show locks' when the reload is hung:
{noformat}
=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked) ===
=== Thread ID: -1208960112 (logger_thread        started at [ 1021] logger.c init_logger())
=== -------------------------------------------------------------------
===
=== Thread ID: -1222329456 (netconsole           started at [ 1326] asterisk.c listener())
=== ---> Lock #0 (loader.c): MUTEX 653 ast_module_reload &reloadlock 0x820dd40 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x19) [0x810e5de]
/usr/sbin/asterisk [0x81083ea]
/usr/sbin/asterisk(ast_module_reload+0x6d) [0x8108066]
/usr/sbin/asterisk [0x80b2691]
/usr/sbin/asterisk(ast_cli_command_full+0x224) [0x80ba44d]
/usr/lib/asterisk/modules/res_clialiases.so [0x34905d]
/usr/sbin/asterisk(ast_cli_command_full+0x224) [0x80ba44d]
/usr/sbin/asterisk(ast_cli_command_multiple_full+0x67) [0x80ba583]
/usr/sbin/asterisk [0x8079e22]
/usr/sbin/asterisk [0x818dae6]
/lib/libpthread.so.0 [0xa39832]
/lib/libc.so.6(clone+0x5e) [0x97945e]
=== ---> Lock #1 (loader.c): MUTEX 672 ast_module_reload &(&module_list)->lock 0x820d0c8 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x19) [0x810e5de]
/usr/sbin/asterisk [0x8105a84]
/usr/sbin/asterisk(ast_module_reload+0x180) [0x8108179]
/usr/sbin/asterisk [0x80b2691]
/usr/sbin/asterisk(ast_cli_command_full+0x224) [0x80ba44d]
/usr/lib/asterisk/modules/res_clialiases.so [0x34905d]
/usr/sbin/asterisk(ast_cli_command_full+0x224) [0x80ba44d]
/usr/sbin/asterisk(ast_cli_command_multiple_full+0x67) [0x80ba583]
/usr/sbin/asterisk [0x8079e22]
/usr/sbin/asterisk [0x818dae6]
/lib/libpthread.so.0 [0xa39832]
/lib/libc.so.6(clone+0x5e) [0x97945e]
=== ---> Waiting for Lock #2 (pbx.c): WRLOCK 9359 ast_wrlock_contexts &conlock 0x8216240 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x19) [0x810e5de]
/usr/sbin/asterisk [0x8121bdd]
/usr/sbin/asterisk(ast_rdlock_contexts+0x32) [0x813d7a5]
/usr/sbin/asterisk [0x812a72c]
/usr/sbin/asterisk(ast_exists_extension+0x55) [0x812c175]
/usr/lib/asterisk/modules/chan_sip.so [0xbb3bcb]
/usr/lib/asterisk/modules/chan_sip.so [0xbd13af]
/usr/lib/asterisk/modules/chan_sip.so [0xbdd050]
/usr/lib/asterisk/modules/chan_sip.so [0xbde334]
/usr/lib/asterisk/modules/chan_sip.so [0xbddad5]
/usr/sbin/asterisk(ast_io_wait+0x14d) [0x8103bad]
/usr/lib/asterisk/modules/chan_sip.so [0xbdf970]
/usr/sbin/asterisk [0x818dae6]
/lib/libpthread.so.0 [0xa39832]
/lib/libc.so.6(clone+0x5e) [0x97945e]
=== -------------------------------------------------------------------
===
{noformat}

I will upload the debug output separately, in this file the issue occurs at 1:31.
Comments:By: Steven Wheeler (swheeler) 2011-07-13 09:16:38.493-0500

Debug output from hung reload.
By: Richard Mudgett (rmudgett) 2011-10-12 16:52:46.590-0500

Is this still happening?
Is that all the "core show locks" output?

The lock being waited upon in the core show locks output is not a WRLOCK so I suspect there might be some memory corruption.  Could you do a reload under valgrind?  (After hours since valgrind *really* slows the machine.)
By: Richard Mudgett (rmudgett) 2011-10-12 16:53:13.261-0500

Your backtrace appears to contain memory corruption and we require valgrind output in order to move this issue forward. Please see https://wiki.asterisk.org/wiki/display/AST/Valgrind for more information about how to produce debugging information. Thanks!


By: Steven Wheeler (swheeler) 2011-10-12 17:04:15.813-0500

We moved this particular PBX up to 1.8.7 and haven't seen this issue since.  However, we are still seeing it very rarely on a couple other PBXs.

The core show locks output I uploaded is all that asterisk gave me.

I will see if I can reproduce this on a different PBX still running 1.6.2 and get you the valgrind output.
By: Richard Mudgett (rmudgett) 2011-10-26 16:30:24.648-0500

Any status update?  Otherwise, I'll just close this issue next week.
By: Steven Wheeler (swheeler) 2011-10-26 16:54:25.532-0500

I am having trouble reproducing the issue.  If you want, you can close this issue and I will reopen it when I can reproduce the problem.
By: Richard Mudgett (rmudgett) 2011-10-26 18:01:35.614-0500

This issue is against a version that is no longer receiving regular maintenance.  I was trying to see if it was still in v1.8 by finding it in the reported version.

Please open a new bug report if this happens in newer versions.