[Home]

Summary:ASTERISK-14337: After a few thousand calls, or at random, Asterisk stops receiving events from the network
Reporter:Private Name (falves11)Labels:
Date Opened:2009-06-18 17:59:47Date Closed:2010-01-05 09:23:03.000-0600
Priority:MinorRegression?Yes
Status:Closed/CompleteComponents:Channels/chan_sip/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) asterisk-dump.pdf
( 1) crash1.txt
( 2) crash2.txt
( 3) crash3.txt
Description:The application is up, you may do "core restart now", "core show channels", etc.,  but no new calls entert and I think no packests are delivered. Other times it is dead and only a "killall asterisk" will restart it.

****** ADDITIONAL INFORMATION ******

I had to go back to SVN-branch-1.6.2-r197190M. That one does not have this issue. It may be possible that also newer versions work, but I only have this one in production.
If I can set some compiler flags to understand the issue, I can upgrade one server to the current verision and reproduce it. I need instructions as to what information gather.
Comments:By: Leif Madsen (lmadsen) 2009-06-24 11:57:03

The standard debugging information applies here in regards to chan_sip:

* sip debug
* sip history
* potentially a wireshark trace
* enable MALLOC_DEBUG and provide "core show locks"
* if Asterisk crashes, provide a backtrace with DONT_OPTIMIZE enabled

This sounds like a deadlock to me.

Additionally, this is not a blocking issue just because it affects your network :)
By: Private Name (falves11) 2009-07-03 12:22:49

It crashes with version SVN-branch-1.6.2-r203608. I uploaded the core trace.
By: Private Name (falves11) 2009-07-08 03:05:06

The latest 3 crash traces are for 204920. They all are in the same function.
By: Private Name (falves11) 2009-07-31 17:02:36

The Parallels engineers have found a bug that takes down asterisk because the server runs out of sockets, and also it degrades the perfornce because it takes more and more time for the processor to find an empty socket. The load on the processor grows over time,
This is what they saw:

"As i can see, the number of UDP sockets are more or less permanently grows:

[root@host-208 ~]# cat /proc/net/sockstat
sockets: used 90645
TCP: inuse 1242 orphan 0 tw 117 alloc 1322 mem 150
UDP: inuse 88881 mem 6744
RAW: inuse 0
FRAG: inuse 0 memory 0



Processors again spend more and more time to get need socket out from 86366 "inuse".However the traffic not growing and much less now:

[root@host-208 ~]# vzstat -l
 5:23pm, up  3:17,  5 users, load average: 24.79, 25.45, 18.39
CTNum 23, procs 1762: R  22, S 1731, D   0, Z   0, T   0, X   9
CPU [ OK ]: CTs 100%, CT0   0%, user  33%, sys   8%, idle  60%, lat(ms)  85/0
Mem [ OK ]: total 128737MB, free 124755MB/0MB (low/high), lat(ms) 7/2
 ZONE0 (DMA): size 9MB, act 0MB, inact 0MB, free 10MB (0/0/0)
 ZONE1 (DMA32): size 2986MB, act 0MB, inact 0MB, free 2789MB (1/1/1)
 ZONE2 (Normal): size 126000MB, act 2756MB, inact 528MB, free 121955MB (43/54/65)
 Mem lat (ms): A0 0, K0 5, U0 0, K1 7, U1 0
 Slab pages: 437MB/437MB (ino 117MB, de 43MB, bh 5MB, pb 35MB)
Swap [ OK ]: tot 2055MB, free 2055MB, in 0.000MB/s, out 0.000MB/s
Net [ OK ]: tot: in  3.275MB/s 15778pkt/s, out  1.876MB/s 8706pkt/s
            lo: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
          eth0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
          eth1: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
          eth2: in  3.275MB/s 15778pkt/s, out  1.876MB/s 8706pkt/s
          eth3: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
          sit0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
    veth9102.0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
    veth9106.0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
    veth9100.0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
    veth9111.0: in  0.000MB/s    0pkt/s, out  0.000MB/s    0pkt/s
Disks [ OK ]: in 0.000MB/s, out 0.480MB/s

[root@host-208 ~]# vztop -n2 -d3
vztop - 17:26:11 up  3:19,  5 users,  load average: 20.27, 23.55, 18.61
Tasks: 902 total,   1 running, 901 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.4% us,  3.3% sy,  0.0% ni, 27.6% id,  0.0% wa,  0.0% hi, 55.7% si
Mem:  131826772k total,  4050364k used, 127776408k free,   111156k buffers
Swap:  2104472k total,        0k used,  2104472k free,  1559484k cached

CTID   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9112 39281 root      18   0  584m 232m  11m S  184  0.2 104:47.71 asterisk
9100 39326 root      18   0  581m 209m  11m S  181  0.2  94:24.69 asterisk
9102 39709 root      18   0  577m 207m  11m S  171  0.2  89:55.23 asterisk
9113 39355 root      18   0  578m 212m  11m S  166  0.2  95:13.56 asterisk
9101 16934 root      18   0  595m 212m  11m S  156  0.2  86:46.97 asterisk
9105 39340 root      18   0  407m  70m  11m S   73  0.1  30:09.99 asterisk
9103 39486 root      18   0  402m  66m  11m S   56  0.1  27:52.46 asterisk
9110 39535 root      18   0  401m  65m  11m S   44  0.1  28:23.30 asterisk
9104 39686 root      18   0  396m  65m  11m S   40  0.1  26:59.71 asterisk
9106 39485 root      18   0  396m  63m  11m S   38  0.0  27:29.80 asterisk
9108 39366 root      18   0  405m  70m  11m S   31  0.1  27:43.83 asterisk
   0 39220 root      15   0 11188 1700  824 R    1  0.0   0:00.08 vztop
   0 51361 root      16   0 11188 1712  816 S    1  0.0   1:15.12 vztop
9114 39062 root      15   0 70824  28m  28m S    1  0.0   0:27.20 opensips
9107 39259 root      18   0  277m  17m  10m S    1  0.0   0:08.53 asterisk
9114 39060 root      15   0 70824  28m  28m S    0  0.0   0:27.18 opensips
   0     1 root      15   0 10344  676  568 S    0  0.0   0:02.85 init
   0     2 root      RT  -5     0    0    0 S    0  0.0   0:00.02 migration/0/0


i guess probably asterisk might leave some sockets open instead of closing them after end of calls/meetings/etc.
By: Digium Subversion (svnbot) 2009-12-10 10:08:40.000-0600

Repository: asterisk
Revision: 234095

U   branches/1.4/channels/chan_sip.c

------------------------------------------------------------------------
r234095 | tilghman | 2009-12-10 10:08:21 -0600 (Thu, 10 Dec 2009) | 9 lines

When we receive no response at all to our INVITE, allow the channel to be destroyed.
(closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
Reported by: falves11
Patches:
      20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
      20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
Tested by: falves11
Review: https://reviewboard.asterisk.org/r/446/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234095
By: Digium Subversion (svnbot) 2009-12-10 10:22:36.000-0600

Repository: asterisk
Revision: 234095

U   branches/1.4/channels/chan_sip.c

------------------------------------------------------------------------
r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines

When we receive no response at all to our INVITE, allow the channel to be destroyed.
(closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
Reported by: falves11, corruptor, dant
Patches:
      20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
      20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
Tested by: falves11
Review: https://reviewboard.asterisk.org/r/446/

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234095
By: Digium Subversion (svnbot) 2009-12-10 10:24:43.000-0600

Repository: asterisk
Revision: 234129

_U  trunk/
U   trunk/channels/chan_sip.c

------------------------------------------------------------------------
r234129 | tilghman | 2009-12-10 10:24:28 -0600 (Thu, 10 Dec 2009) | 16 lines

Merged revisions 234095 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
 
 When we receive no response at all to our INVITE, allow the channel to be destroyed.
 (closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
  Reported by: falves11, corruptor, dant
  Patches:
        20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
        20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
  Tested by: falves11
 Review: https://reviewboard.asterisk.org/r/446/
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234129
By: Digium Subversion (svnbot) 2009-12-10 10:30:35.000-0600

Repository: asterisk
Revision: 234131

_U  branches/1.6.0/
U   branches/1.6.0/channels/chan_sip.c

------------------------------------------------------------------------
r234131 | tilghman | 2009-12-10 10:30:25 -0600 (Thu, 10 Dec 2009) | 23 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
   
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
    Reported by: falves11, corruptor, dant
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234131
By: Digium Subversion (svnbot) 2009-12-10 10:30:50.000-0600

Repository: asterisk
Revision: 234132

_U  branches/1.6.1/
U   branches/1.6.1/channels/chan_sip.c

------------------------------------------------------------------------
r234132 | tilghman | 2009-12-10 10:30:39 -0600 (Thu, 10 Dec 2009) | 23 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
   
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
    Reported by: falves11, corruptor, dant
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234132
By: Digium Subversion (svnbot) 2009-12-10 10:38:55.000-0600

Repository: asterisk
Revision: 234133

_U  branches/1.6.2/
U   branches/1.6.2/channels/chan_sip.c

------------------------------------------------------------------------
r234133 | tilghman | 2009-12-10 10:38:45 -0600 (Thu, 10 Dec 2009) | 23 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
   
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576, closes issue ASTERISK-14653, closes issue ASTERISK-15161, closes issue ASTERISK-14337, issue ASTERISK-14500)
    Reported by: falves11, corruptor, dant
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234133
By: Digium Subversion (svnbot) 2009-12-10 10:53:52.000-0600

Repository: asterisk
Revision: 234095

U   branches/1.4/channels/chan_sip.c

------------------------------------------------------------------------
r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 17 lines

When we receive no response at all to our INVITE, allow the channel to be destroyed.
(closes issue ASTERISK-14576)
Reported by: falves11
Patches:
      20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
      20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
Tested by: falves11
Review: https://reviewboard.asterisk.org/r/446/
(closes issue ASTERISK-14653)
Reported by: dant
(closes issue ASTERISK-15161)
Reported by: corruptor
(closes issue ASTERISK-14337)
Reported by: falves11
(issue ASTERISK-14500)
Reported by: lftsy

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234095
By: Digium Subversion (svnbot) 2009-12-10 10:55:20.000-0600

Repository: asterisk
Revision: 234129

_U  trunk/
U   trunk/channels/chan_sip.c

------------------------------------------------------------------------
r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 24 lines

Merged revisions 234095 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
 
 When we receive no response at all to our INVITE, allow the channel to be destroyed.
 (closes issue ASTERISK-14576)
  Reported by: falves11
  Patches:
        20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
        20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
  Tested by: falves11
 Review: https://reviewboard.asterisk.org/r/446/
 (closes issue ASTERISK-14653)
 Reported by: dant
 (closes issue ASTERISK-15161)
 Reported by: corruptor
 (closes issue ASTERISK-14337)
 Reported by: falves11
 (issue ASTERISK-14500)
 Reported by: lftsy
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234129
By: Digium Subversion (svnbot) 2009-12-10 10:56:26.000-0600

Repository: asterisk
Revision: 234131

_U  branches/1.6.0/
U   branches/1.6.0/channels/chan_sip.c

------------------------------------------------------------------------
r234131 | tilghman | 2009-12-10 10:30:22 -0600 (Thu, 10 Dec 2009) | 29 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576)
    Reported by: falves11
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
   (closes issue ASTERISK-14653)
   Reported by: dant
   (closes issue ASTERISK-15161)
   Reported by: corruptor
   (closes issue ASTERISK-14337)
   Reported by: falves11
   (issue ASTERISK-14500)
   Reported by: lftsy
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234131
By: Digium Subversion (svnbot) 2009-12-10 10:57:19.000-0600

Repository: asterisk
Revision: 234132

_U  branches/1.6.1/
U   branches/1.6.1/channels/chan_sip.c

------------------------------------------------------------------------
r234132 | tilghman | 2009-12-10 10:30:32 -0600 (Thu, 10 Dec 2009) | 31 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
   
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576)
    Reported by: falves11
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
   (closes issue ASTERISK-14653)
   Reported by: dant
   (closes issue ASTERISK-15161)
   Reported by: corruptor
   (closes issue ASTERISK-14337)
   Reported by: falves11
   (issue ASTERISK-14500)
   Reported by: lftsy
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234132
By: Digium Subversion (svnbot) 2009-12-10 10:57:48.000-0600

Repository: asterisk
Revision: 234133

_U  branches/1.6.2/
U   branches/1.6.2/channels/chan_sip.c

------------------------------------------------------------------------
r234133 | tilghman | 2009-12-10 10:30:40 -0600 (Thu, 10 Dec 2009) | 31 lines

Merged revisions 234129 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r234129 | tilghman | 2009-12-10 10:24:26 -0600 (Thu, 10 Dec 2009) | 16 lines
 
 Merged revisions 234095 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r234095 | tilghman | 2009-12-10 10:08:20 -0600 (Thu, 10 Dec 2009) | 9 lines
   
   When we receive no response at all to our INVITE, allow the channel to be destroyed.
   (closes issue ASTERISK-14576)
    Reported by: falves11
    Patches:
          20091209__issue15627__1.6.0.diff.txt uploaded by tilghman (license 14)
          20091209__issue15627__1.4.diff.txt uploaded by tilghman (license 14)
    Tested by: falves11
   Review: https://reviewboard.asterisk.org/r/446/
   (closes issue ASTERISK-14653)
   Reported by: dant
   (closes issue ASTERISK-15161)
   Reported by: corruptor
   (closes issue ASTERISK-14337)
   Reported by: falves11
   (issue ASTERISK-14500)
   Reported by: lftsy
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=234133