[Home]

Summary:ASTERISK-13659: [patch] safe_asterisk can get multiple instances if killproc escalates to SIGKILL in service asterisk restart
Reporter:David Woolley (davidw)Labels:
Date Opened:2009-02-26 12:37:05.000-0600Date Closed:2009-11-02 11:22:30.000-0600
Priority:MinorRegression?No
Status:Closed/CompleteComponents:General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20091022__issue14562.diff.txt
( 1) 20091030__Issue14562_diff.txt
Description:Weird behaviour was observed in which we got Remote UNIX connection and disconnections continuously repeated.  It turned out that we had two copies of safe_asterisk running.

Although I didn't do this myself, I believe this was the result of using "service asterisk restart" on an asterisk that was in distress (deadlock, at least).  I think what happened is that the killproc escalated to SIGKILL, which was not recognized by the original safe_asterisk, so it continued to run.  Meanwhile the new safe_asterisk was started, but both checks for a running asterisk failed, either because it was too much in distress, or because the console listener had shut down in response to killproc's SIGTERM.

****** ADDITIONAL INFORMATION ******

Linux centos 2.6.18-92.1.13.el5 #1 SMP Wed Sep 24 19:33:52 EDT 2008 i686 i686 i3
86 GNU/Linux
Comments:By: Leif Madsen (lmadsen) 2009-07-01 08:29:57

I've seen this happen before too. I'm going to acknowledge the issue, and assign it to target version. Thanks!
By: Tilghman Lesher (tilghman) 2009-10-22 17:45:13

Patch uploaded, ready for testing to verify that it solves the issue!
By: Tilghman Lesher (tilghman) 2009-10-27 15:09:08

davidw: I need feedback on this patch or it will be closed.
By: David Woolley (davidw) 2009-10-28 06:25:57

I'll see what I can do, but the person who actually triggered the fault is on leave at the moment.
By: David Woolley (davidw) 2009-10-30 10:15:32

Looks OK, except that it messes up the progress messages.  Will upload revised version.

Tried using kill -STOP on asterisk to simulate deadlock.  Without the patch the result was:

[root@centos init.d]# kill -STOP 756
[root@centos init.d]# service asterisk restart
Shutting down asterisk: /usr/sbin/safe_asterisk: line 146:   756 Killed        
        nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /de
v/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 137
Asterisk exited on signal 9.
Automatically restarting Asterisk.
[  OK  ]
Starting asterisk: [  OK  ]
[root@centos init.d]# mpg123: no process killed
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
mpg123: no process killed
Asterisk ended with exit status 1


With the patch:
[root@centos init.d]# ps -e | grep asterisk
1390 pts/1    00:00:00 safe_asterisk
1400 pts/1    00:00:00 asterisk
[root@centos init.d]# kill -STOP 1400
[root@centos init.d]# service asterisk restart
[  OK  ] down asterisk: [  OK  ]
Starting asterisk: [  OK  ]
[root@centos init.d]# ps -e | grep asterisk
1544 pts/1    00:00:00 safe_asterisk
1554 pts/1    00:00:00 asterisk

As modified:
[root@centos init.d]# ps  -e | grep asterisk
1825 pts/1    00:00:00 safe_asterisk
1831 pts/1    00:00:01 asterisk
[root@centos init.d]# kill -STOP 1831
[root@centos init.d]# service asterisk restart
Stopping safe_asterisk: [  OK  ]
Shutting down asterisk: [  OK  ]
Starting asterisk: [  OK  ]
[root@centos init.d]# ps -e | grep asterisk
2402 pts/1    00:00:00 safe_asterisk
2412 pts/1    00:00:02 asterisk
By: Digium Subversion (svnbot) 2009-11-02 11:19:38.000-0600

Repository: asterisk
Revision: 226811

U   branches/1.4/contrib/init.d/rc.redhat.asterisk

------------------------------------------------------------------------
r226811 | tilghman | 2009-11-02 11:19:37 -0600 (Mon, 02 Nov 2009) | 8 lines

Don't allow two separate instances of safe_asterisk when restarting from the init script.
(closes issue ASTERISK-13659)
Reported by: davidw
Patches:
      Initially 20091022__issue14562.diff.txt uploaded by tilghman (license 14)
      Modified to 20091030__Issue14562_diff.txt uploaded by davidw (license 780)
Tested by: davidw

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=226811
By: Digium Subversion (svnbot) 2009-11-02 11:20:49.000-0600

Repository: asterisk
Revision: 226812

_U  trunk/
U   trunk/contrib/init.d/rc.redhat.asterisk

------------------------------------------------------------------------
r226812 | tilghman | 2009-11-02 11:20:48 -0600 (Mon, 02 Nov 2009) | 15 lines

Merged revisions 226811 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r226811 | tilghman | 2009-11-02 11:14:20 -0600 (Mon, 02 Nov 2009) | 8 lines
 
 Don't allow two separate instances of safe_asterisk when restarting from the init script.
 (closes issue ASTERISK-13659)
  Reported by: davidw
  Patches:
        Initially 20091022__issue14562.diff.txt uploaded by tilghman (license 14)
        Modified to 20091030__Issue14562_diff.txt uploaded by davidw (license 780)
  Tested by: davidw
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=226812
By: Digium Subversion (svnbot) 2009-11-02 11:22:13.000-0600

Repository: asterisk
Revision: 226813

_U  branches/1.6.0/
U   branches/1.6.0/contrib/init.d/rc.redhat.asterisk

------------------------------------------------------------------------
r226813 | tilghman | 2009-11-02 11:22:13 -0600 (Mon, 02 Nov 2009) | 22 lines

Merged revisions 226812 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r226812 | tilghman | 2009-11-02 11:15:31 -0600 (Mon, 02 Nov 2009) | 15 lines
 
 Merged revisions 226811 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r226811 | tilghman | 2009-11-02 11:14:20 -0600 (Mon, 02 Nov 2009) | 8 lines
   
   Don't allow two separate instances of safe_asterisk when restarting from the init script.
   (closes issue ASTERISK-13659)
    Reported by: davidw
    Patches:
          Initially 20091022__issue14562.diff.txt uploaded by tilghman (license 14)
          Modified to 20091030__Issue14562_diff.txt uploaded by davidw (license 780)
    Tested by: davidw
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=226813
By: Digium Subversion (svnbot) 2009-11-02 11:22:21.000-0600

Repository: asterisk
Revision: 226814

_U  branches/1.6.1/
U   branches/1.6.1/contrib/init.d/rc.redhat.asterisk

------------------------------------------------------------------------
r226814 | tilghman | 2009-11-02 11:22:20 -0600 (Mon, 02 Nov 2009) | 22 lines

Merged revisions 226812 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r226812 | tilghman | 2009-11-02 11:15:31 -0600 (Mon, 02 Nov 2009) | 15 lines
 
 Merged revisions 226811 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r226811 | tilghman | 2009-11-02 11:14:20 -0600 (Mon, 02 Nov 2009) | 8 lines
   
   Don't allow two separate instances of safe_asterisk when restarting from the init script.
   (closes issue ASTERISK-13659)
    Reported by: davidw
    Patches:
          Initially 20091022__issue14562.diff.txt uploaded by tilghman (license 14)
          Modified to 20091030__Issue14562_diff.txt uploaded by davidw (license 780)
    Tested by: davidw
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=226814
By: Digium Subversion (svnbot) 2009-11-02 11:22:29.000-0600

Repository: asterisk
Revision: 226815

_U  branches/1.6.2/
U   branches/1.6.2/contrib/init.d/rc.redhat.asterisk

------------------------------------------------------------------------
r226815 | tilghman | 2009-11-02 11:22:29 -0600 (Mon, 02 Nov 2009) | 22 lines

Merged revisions 226812 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r226812 | tilghman | 2009-11-02 11:15:31 -0600 (Mon, 02 Nov 2009) | 15 lines
 
 Merged revisions 226811 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r226811 | tilghman | 2009-11-02 11:14:20 -0600 (Mon, 02 Nov 2009) | 8 lines
   
   Don't allow two separate instances of safe_asterisk when restarting from the init script.
   (closes issue ASTERISK-13659)
    Reported by: davidw
    Patches:
          Initially 20091022__issue14562.diff.txt uploaded by tilghman (license 14)
          Modified to 20091030__Issue14562_diff.txt uploaded by davidw (license 780)
    Tested by: davidw
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=226815