[Home]

Summary:ASTERISK-14197: [patch] #exec script can't access manager on first asterisk load
Reporter:Philippe Lindheimer (p_lindheimer)Labels:
Date Opened:2009-05-24 12:32:03Date Closed:2009-06-04 09:53:59
Priority:BlockerRegression?Yes
Status:Closed/CompleteComponents:Core/PBX
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 05292009_15189_idea_1.diff
( 1) 06032009_15189_deferred_reloads.diff
Description:In earlier versions of Asterisk (1.4.21.1 tested, also 1.2.x I believe), a '#exec' directive in extensions.conf can run a script that can then access the manager which is required to get to ASTDB to obtain information to generate dialplan.

A fix to bug ASTERISK-12961 (revision 151905) has broken this.

****** STEPS TO REPRODUCE ******

Insert a #exec in extensions.conf that runs a script that connects to the manager
Have the script print out debug information if it can't connect to the manager
start Asterisk

result: it can't connect

type reload at the CLI

result: now it can connect

****** ADDITIONAL INFORMATION ******

This capability is used by FreePBX, a script called generate_hints.php

The purpose of the script is to dynamically generate hints for extensions based on the state information contained in the ASTDB. This needs to occur dynamically because it is possible for 'users' to dynamically log into and out of a device, turning it into their extension. (Hot Desk Feature).

Given that this was checked in several months ago, it is present in recent tarball releases.
Comments:By: Tilghman Lesher (tilghman) 2009-05-27 02:27:05

One workaround would be to use the "asterisk -rx 'database show'" method of accessing astdb.
By: Philippe Lindheimer (p_lindheimer) 2009-05-27 09:08:36

It would be preferable to not require a workaround if there are other solutions that can address ASTERISK-1367778 so this regression can be fixed. Although currently only accessing AstDB in this exec script, there are potentials that other information from the manager might need to be accessed at some point.

Not to overly discuss ASTERISK-1367778 in this bug thread, but does the fix that was put in place for that bug solve all race conditions? (e.g. what if a reload is executed from the CLI, and then while reloading, another one is executed from the AMI? Is that a different code path?)
By: Sean Bright (seanbright) 2009-05-28 23:04:40

I've uploaded an initial patch.  This is just a proof of concept at this point but I wanted to test the idea.  Please give this a test and report the results.

With this, pbx_config.so will load, loads extensions.conf, and registers a callback that will be called once the manager loads.  pbx_config then does a reload.  As I said, this is far from optimal, but it's a start.
By: Tilghman Lesher (tilghman) 2009-05-29 00:01:22

Another way to approach this is to add the ast_mutex_timedlock call, similar to the way I added ast_rwlock_timedwrlock and ast_rwlock_timedrdlock, and use that in ast_module_reload to avoid the previous deadlock situation.  This means that a reload might fail, but that's preferable to the alternatives (either a deadlock or a module load failing).


By: Philippe Lindheimer (p_lindheimer) 2009-05-29 09:03:39

tested the provided patch and it looks like it solves the problem. Can't say anything about stress testing, but the manager connection is successful on initial startup from the #exec script and the hints are all generated.
By: Sean Bright (seanbright) 2009-06-03 12:58:01

p_lindheimer,

I've uploaded a new patch (06032009_15189_deferred_reloads.diff) that I would like you to test against 1.4.  It should allow your generate_hints.php script to run without deadlocking and without an unnecessary reload.  Please test and report back your results.

Thanks.
By: Philippe Lindheimer (p_lindheimer) 2009-06-03 13:41:00

looks good, hints are appearing (didn't have to remember to reload Asterisk after make install this time, it's required to test this :-) )
By: Digium Subversion (svnbot) 2009-06-04 09:14:59

Repository: asterisk
Revision: 199022

U   branches/1.4/include/asterisk.h
U   branches/1.4/main/asterisk.c
U   branches/1.4/main/loader.c

------------------------------------------------------------------------
r199022 | seanbright | 2009-06-04 09:14:58 -0500 (Thu, 04 Jun 2009) | 39 lines

Safely handle AMI connections/reload requests that occur during startup.

During asterisk startup, a lock on the list of modules is obtained by the
primary thread while each module is initialized.  Issue 13778 pointed out a
problem with this approach, however.  Because the AMI is loaded before other
modules, it is possible for a module reload to be issued by a connected client
(via Action: Command), causing a deadlock.

The resolution for 13778 was to move initialization of the manager to happen
after the other modules had already been lodaded.  While this fixed this
particular issue, it caused a problem for users (like FreePBX) who call AMI
scripts via an #exec in a configuration file (See issue 15189).

The solution I have come up with is to defer any reload requests that come in
until after the server is fully booted.  When a call comes in to
ast_module_reload (from wherever) before we are fully booted, the request is
added to a queue of pending requests.  Once we are done booting up, we then
execute these deferred requests in turn.

Note that I have tried to make this a bit more intelligent in that it will not
queue up more than 1 request for the same module to be reloaded, and if a
general reload request comes in ('module reload') the queue is flushed and we
only issue a single deferred reload for the entire system.

As for how this will impact existing installations - Before 13778, a reload
issued before module initialization was completed would result in a deadlock.
After 13778, you simply couldn't connect to the manager during startup (which
causes problems with #exec-that-calls-AMI configuration files).  I believe this
is a good general purpose solution that won't negatively impact existing
installations.

(closes issue ASTERISK-14197)
Reported by: p_lindheimer
Patches:
     06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
Tested by: p_lindheimer, seanbright

Review: https://reviewboard.asterisk.org/r/272/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199022
By: Digium Subversion (svnbot) 2009-06-04 09:17:21

Repository: asterisk
Revision: 199022

U   branches/1.4/include/asterisk.h
U   branches/1.4/main/asterisk.c
U   branches/1.4/main/loader.c

------------------------------------------------------------------------
r199022 | seanbright | 2009-06-04 09:14:57 -0500 (Thu, 04 Jun 2009) | 40 lines

Safely handle AMI connections/reload requests that occur during startup.

During asterisk startup, a lock on the list of modules is obtained by the
primary thread while each module is initialized.  Issue 13778 pointed out a
problem with this approach, however.  Because the AMI is loaded before other
modules, it is possible for a module reload to be issued by a connected client
(via Action: Command), causing a deadlock.

The resolution for 13778 was to move initialization of the manager to happen
after the other modules had already been lodaded.  While this fixed this
particular issue, it caused a problem for users (like FreePBX) who call AMI
scripts via an #exec in a configuration file (See issue 15189).

The solution I have come up with is to defer any reload requests that come in
until after the server is fully booted.  When a call comes in to
ast_module_reload (from wherever) before we are fully booted, the request is
added to a queue of pending requests.  Once we are done booting up, we then
execute these deferred requests in turn.

Note that I have tried to make this a bit more intelligent in that it will not
queue up more than 1 request for the same module to be reloaded, and if a
general reload request comes in ('module reload') the queue is flushed and we
only issue a single deferred reload for the entire system.

As for how this will impact existing installations - Before 13778, a reload
issued before module initialization was completed would result in a deadlock.
After 13778, you simply couldn't connect to the manager during startup (which
causes problems with #exec-that-calls-AMI configuration files).  I believe this
is a good general purpose solution that won't negatively impact existing
installations.

(closes issue ASTERISK-14197)
(closes issue ASTERISK-12961)
Reported by: p_lindheimer
Patches:
     06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
Tested by: p_lindheimer, seanbright

Review: https://reviewboard.asterisk.org/r/272/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199022
By: Digium Subversion (svnbot) 2009-06-04 09:31:25

Repository: asterisk
Revision: 199051

_U  trunk/
U   trunk/include/asterisk/_private.h
U   trunk/main/asterisk.c
U   trunk/main/loader.c

------------------------------------------------------------------------
r199051 | seanbright | 2009-06-04 09:31:24 -0500 (Thu, 04 Jun 2009) | 47 lines

Merged revisions 199022 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
 
 Safely handle AMI connections/reload requests that occur during startup.
 
 During asterisk startup, a lock on the list of modules is obtained by the
 primary thread while each module is initialized.  Issue 13778 pointed out a
 problem with this approach, however.  Because the AMI is loaded before other
 modules, it is possible for a module reload to be issued by a connected client
 (via Action: Command), causing a deadlock.
 
 The resolution for 13778 was to move initialization of the manager to happen
 after the other modules had already been lodaded.  While this fixed this
 particular issue, it caused a problem for users (like FreePBX) who call AMI
 scripts via an #exec in a configuration file (See issue 15189).
 
 The solution I have come up with is to defer any reload requests that come in
 until after the server is fully booted.  When a call comes in to
 ast_module_reload (from wherever) before we are fully booted, the request is
 added to a queue of pending requests.  Once we are done booting up, we then
 execute these deferred requests in turn.
 
 Note that I have tried to make this a bit more intelligent in that it will not
 queue up more than 1 request for the same module to be reloaded, and if a
 general reload request comes in ('module reload') the queue is flushed and we
 only issue a single deferred reload for the entire system.
 
 As for how this will impact existing installations - Before 13778, a reload
 issued before module initialization was completed would result in a deadlock.
 After 13778, you simply couldn't connect to the manager during startup (which
 causes problems with #exec-that-calls-AMI configuration files).  I believe this
 is a good general purpose solution that won't negatively impact existing
 installations.
 
 (closes issue ASTERISK-14197)
 (closes issue ASTERISK-12961)
 Reported by: p_lindheimer
 Patches:
       06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
 Tested by: p_lindheimer, seanbright
 
 Review: https://reviewboard.asterisk.org/r/272/
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199051
By: Digium Subversion (svnbot) 2009-06-04 09:53:50

Repository: asterisk
Revision: 199052

_U  branches/1.6.0/
U   branches/1.6.0/include/asterisk/_private.h
U   branches/1.6.0/main/asterisk.c
U   branches/1.6.0/main/loader.c

------------------------------------------------------------------------
r199052 | seanbright | 2009-06-04 09:53:49 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199052
By: Digium Subversion (svnbot) 2009-06-04 09:53:54

Repository: asterisk
Revision: 199053

_U  branches/1.6.1/
U   branches/1.6.1/include/asterisk/_private.h
U   branches/1.6.1/main/asterisk.c
U   branches/1.6.1/main/loader.c

------------------------------------------------------------------------
r199053 | seanbright | 2009-06-04 09:53:54 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199053
By: Digium Subversion (svnbot) 2009-06-04 09:53:58

Repository: asterisk
Revision: 199054

_U  branches/1.6.2/
U   branches/1.6.2/include/asterisk/_private.h
U   branches/1.6.2/main/asterisk.c
U   branches/1.6.2/main/loader.c

------------------------------------------------------------------------
r199054 | seanbright | 2009-06-04 09:53:58 -0500 (Thu, 04 Jun 2009) | 54 lines

Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47 lines
 
 Merged revisions 199022 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) | 40 lines
   
   Safely handle AMI connections/reload requests that occur during startup.
   
   During asterisk startup, a lock on the list of modules is obtained by the
   primary thread while each module is initialized.  Issue 13778 pointed out a
   problem with this approach, however.  Because the AMI is loaded before other
   modules, it is possible for a module reload to be issued by a connected client
   (via Action: Command), causing a deadlock.
   
   The resolution for 13778 was to move initialization of the manager to happen
   after the other modules had already been lodaded.  While this fixed this
   particular issue, it caused a problem for users (like FreePBX) who call AMI
   scripts via an #exec in a configuration file (See issue 15189).
   
   The solution I have come up with is to defer any reload requests that come in
   until after the server is fully booted.  When a call comes in to
   ast_module_reload (from wherever) before we are fully booted, the request is
   added to a queue of pending requests.  Once we are done booting up, we then
   execute these deferred requests in turn.
   
   Note that I have tried to make this a bit more intelligent in that it will not
   queue up more than 1 request for the same module to be reloaded, and if a
   general reload request comes in ('module reload') the queue is flushed and we
   only issue a single deferred reload for the entire system.
   
   As for how this will impact existing installations - Before 13778, a reload
   issued before module initialization was completed would result in a deadlock.
   After 13778, you simply couldn't connect to the manager during startup (which
   causes problems with #exec-that-calls-AMI configuration files).  I believe this
   is a good general purpose solution that won't negatively impact existing
   installations.
   
   (closes issue ASTERISK-14197)
   (closes issue ASTERISK-12961)
   Reported by: p_lindheimer
   Patches:
         06032009_15189_deferred_reloads.diff uploaded by seanbright (license 71)
   Tested by: p_lindheimer, seanbright
   
   Review: https://reviewboard.asterisk.org/r/272/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=199054