[Home]

Summary:ASTERISK-12961: asterisk blocked at startup between main/asterisk.c/loader.c/load_modules and manager.c/loader.c/ast_module_reload
Reporter:sebastien (hotsblanc)Labels:
Date Opened:2008-10-24 15:18:12Date Closed:2009-06-04 09:53:59
Priority:MajorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:- using asterisk-1.4.11 (there is no easy way for me right now to update to SVN header for 1.4 or 1.6 trunk)
- OS is Linux 2.6.18_pro500-440epx_eval (ppc amcc)

when starting an AMI client right after asterisk and issuing an AMI reload, asterisk gets locked and cannot for instance process any SIP or AMI message.

we do launch an AMI reload at startup to update few asterisk configuration files on the fly after getting some dynamic information from external processes (and asterisk gets started just before our AMI client).

- putting traces in the code, at startup main/asterisk.c calls

loader.c/load_modules(0)

which locks the module_list with

AST_LIST_LOCK(&module_list);

[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack for thread 805449056
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 0 /usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 1 /usr/sbin/asterisk(load_modules+0x50) [0x10067b90]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 2 /usr/sbin/asterisk(main+0x1928) [0x1002790c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 3 /lib/libc.so.6 [0xfb99134]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 4 /lib/libc.so.6 [0xfb9935c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK load_modules
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK module_register

before the load_modules method can finish, the AMI client 'reload' call ends up calling loader.c/ast_module_reload which locks the same linked module list head object:

AST_LIST_LOCK(&module_list);

[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack for thread 806479024
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 0 /usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 1 /usr/sbin/asterisk(ast_module_reload+0x8c) [0x10066810]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 2 /usr/sbin/asterisk [0x10042d78]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 3 /usr/sbin/asterisk(ast_cli_command+0xb0) [0x10046ae4]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 4 /usr/sbin/asterisk [0x10072474]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 5 /usr/sbin/asterisk [0x10073290]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 6 /usr/sbin/asterisk [0x10074750]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 7 /usr/sbin/asterisk [0x10074798]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 8 /usr/sbin/asterisk [0x100b45cc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 9 /lib/libpthread.so.0 [0xffa9758]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === ast_module_reload before LOCK

the list_lock call in loader.c/ast_module_reload method never returns
and loader.c/load_modules method never returns either, so none of these 2 methods end up calling

AST_LIST_UNLOCK(&module_list);

this happens to me 4 out of 5 times when starting my AMI client after asterisk with no real dependency between the 2 startup but the fact that the asterisk process is up linux wise.

work-around: call init_manager() after load_modules(0) in main/asterisk.c so that my AMI client fails creating the TCP manager socket and keeps on trying; that way whenever the client TCP socket gets created and AMI reload gets called, load_modules method call is ensured to have ended.

or ... would there be another command I could issue first to know that asterisk is fully loaded, .e.g main code done and asterisk in main loop?

****** ADDITIONAL INFORMATION ******

this is major to me because asterisk does not process any SIP or AMI message from then on; I could always put a sleep call in my client code before creating the AMI socket but this does not seem very safe either
Comments:By: Russell Bryant (russell) 2008-10-24 16:50:17

This issue has already been addressed in later versions
By: Russell Bryant (russell) 2008-10-25 05:30:59

I'm reopening this issue because I originally misunderstood what operation was being done from the manager interface.  I think there is a reasonable fix that can be made ...
By: Digium Subversion (svnbot) 2008-10-25 05:48:10

Repository: asterisk
Revision: 151905

U   branches/1.4/main/asterisk.c

------------------------------------------------------------------------
r151905 | russell | 2008-10-25 05:48:09 -0500 (Sat, 25 Oct 2008) | 8 lines

Move AMI initialization to occur after loading modules.  This prevents a
deadlock when someone tries to initiate a module reload from the AMI just
as Asterisk is starting.

(closes issue ASTERISK-12961)
Reported by: hotsblanc
Fix suggested by hotsblanc

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=151905
By: Sean Bright (seanbright) 2009-06-01 10:59:00

Reopening as this is causing some problems with #exec'ing during startup.
By: Sean Bright (seanbright) 2009-06-03 13:00:26

hotsblanc,

I know it has been a while since this bug was addressed, but if you could try the second patch in issue 15189 (the one called 06032009_15189_deferred_reloads.diff) and report back if that solves the deadlock issue you were seeing, that would be great.

Thanks.
By: Digium Subversion (svnbot) 2009-06-04 09:17:22

Repository: asterisk
Revision: 199022

U   branches/1.4/include/asterisk.h
U   branches/1.4/main/asterisk.c
U   branches/1.4/main/loader.c

------------------------------------------------------------------------
r199022 | seanbright | 2009-06-04 09:14:57 -0500 (Thu, 04 Jun 2009) | 40 lines

Safely handle AMI connections/reload requests that occur during startup.

During asterisk startup, a lock on the list of modules is obtained by the
primary thread while each module is initialized.  Issue 13778 pointed out a
problem with this approach, however.  Because the AMI is loaded before other
modules, it is possible for a module reload to be issued by a connected client
(via Action: Command), causing a deadlock.

The resolution for 13778 was to move initialization of the manager to happen
after the other modules had already been lodaded.  While this fixed this
particular issue, it caused a problem for users (like FreePBX) who call AMI
scripts via an #exec in a configuration file (See issue 15189).

The solution I have come up with is to defer any reload requests that come in
until after the server is fully booted.  When a call comes in to
ast_module_reload (from wherever) before we are fully booted, the request is
added to a queue of pending requests.  Once we are done booting up, we then
execute these deferred requests in turn.

Note that I have tried to make this a bit more intelligent in that it will not
queue up more than 1 request for the same module to be reloaded, and if a
general reload request comes in ('module reload') the queue is flushed and we
only issue a single deferred reload for the entire system.

As for how this will impact existing installations - Before 13778, a reload
issued before module initialization was completed would result in a deadlock.
After 13778, you simply couldn't connect to the manager during startup (which
causes problems with #exec-that-calls-AMI configuration files).  I believe this
is a good general purpose solution that won't negatively impact existing
installations.

(closes issue ASTERISK-14197)
(closes issue ASTERISK-12961)
Reported by: p_lindheimer
Patches:
     06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
Tested by: p_lindheimer, seanbright

Review: https://reviewboard.asterisk.org/r/272/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199022
By: Digium Subversion (svnbot) 2009-06-04 09:31:25

Repository: asterisk
Revision: 199051

_U  trunk/
U   trunk/include/asterisk/_private.h
U   trunk/main/asterisk.c
U   trunk/main/loader.c

------------------------------------------------------------------------
r199051 | seanbright | 2009-06-04 09:31:24 -0500 (Thu, 04 Jun 2009) | 47 lines

Merged revisions 199022 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
 
 Safely handle AMI connections/reload requests that occur during startup.
 
 During asterisk startup, a lock on the list of modules is obtained by the
 primary thread while each module is initialized.  Issue 13778 pointed out a
 problem with this approach, however.  Because the AMI is loaded before other
 modules, it is possible for a module reload to be issued by a connected client
 (via Action: Command), causing a deadlock.
 
 The resolution for 13778 was to move initialization of the manager to happen
 after the other modules had already been lodaded.  While this fixed this
 particular issue, it caused a problem for users (like FreePBX) who call AMI
 scripts via an #exec in a configuration file (See issue 15189).
 
 The solution I have come up with is to defer any reload requests that come in
 until after the server is fully booted.  When a call comes in to
 ast_module_reload (from wherever) before we are fully booted, the request is
 added to a queue of pending requests.  Once we are done booting up, we then
 execute these deferred requests in turn.
 
 Note that I have tried to make this a bit more intelligent in that it will not
 queue up more than 1 request for the same module to be reloaded, and if a
 general reload request comes in ('module reload') the queue is flushed and we
 only issue a single deferred reload for the entire system.
 
 As for how this will impact existing installations - Before 13778, a reload
 issued before module initialization was completed would result in a deadlock.
 After 13778, you simply couldn't connect to the manager during startup (which
 causes problems with #exec-that-calls-AMI configuration files).  I believe this
 is a good general purpose solution that won't negatively impact existing
 installations.
 
 (closes issue ASTERISK-14197)
 (closes issue ASTERISK-12961)
 Reported by: p_lindheimer
 Patches:
       06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
 Tested by: p_lindheimer, seanbright
 
 Review: https://reviewboard.asterisk.org/r/272/
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199051
By: Digium Subversion (svnbot) 2009-06-04 09:53:50

Repository: asterisk
Revision: 199052

_U  branches/1.6.0/
U   branches/1.6.0/include/asterisk/_private.h
U   branches/1.6.0/main/asterisk.c
U   branches/1.6.0/main/loader.c

------------------------------------------------------------------------
r199052 | seanbright | 2009-06-04 09:53:49 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199052
By: Digium Subversion (svnbot) 2009-06-04 09:53:55

Repository: asterisk
Revision: 199053

_U  branches/1.6.1/
U   branches/1.6.1/include/asterisk/_private.h
U   branches/1.6.1/main/asterisk.c
U   branches/1.6.1/main/loader.c

------------------------------------------------------------------------
r199053 | seanbright | 2009-06-04 09:53:54 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199053
By: Digium Subversion (svnbot) 2009-06-04 09:53:59

Repository: asterisk
Revision: 199054

_U  branches/1.6.2/
U   branches/1.6.2/include/asterisk/_private.h
U   branches/1.6.2/main/asterisk.c
U   branches/1.6.2/main/loader.c

------------------------------------------------------------------------
r199054 | seanbright | 2009-06-04 09:53:58 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199054