[Home]

Summary:ASTERISK-07449: Asterisk crashing in meetme several times a day
Reporter:sedwards (sedwards)Labels:
Date Opened:2006-08-02 22:16:18Date Closed:2007-06-30 09:20:04
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Applications/app_meetme
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20060805__bug7647__2.diff.txt
( 1) 26370-bt-full-b
( 2) 7647.patch.rev3
( 3) bt-43977
( 4) bt-8732
( 5) bt-9967
( 6) bt-full-43977
( 7) core.13871-bt
( 8) core.14223-bt
( 9) core.16329-bt-full
(10) core.17034-bt
(11) core.20647-bt.gz
(12) core.26173-bt.gz
(13) core.3284
(14) core.4610-bt
(15) core.8764-bt.gz
Description:Asterisk 1.2.10 receiving SIP calls from a Tekelec-7000/r4.0.

Sometimes it crashes, sometimes it accumulates hundreds of channels that aren't really there and stops accepting new calls.

****** ADDITIONAL INFORMATION ******

I use a "home grown" application to play a file at the start of a call and a modified version of cdr_addon_mysql, but I don't suspect these becuase the crash is always in meetme. If you want the source to these applications, just ask :)

I also use several AGI's I've written, but I don't suspect them. Again, if you want the source, just ask :)
Comments:By: sedwards (sedwards) 2006-08-02 22:23:34

I've attached a couple additional "bt's" just in case they can help.
By: Russell Bryant (russell) 2006-08-05 01:18:56

Do all of the backtraces show the crash occuring at this line?

#0  0x002e493b in conf_run (chan=0x9ba6cd8, conf=0x9afe5d0, confflags=135296) at app_meetme.c:873
873 user->user_no = conf->lastuser->user_no + 1;
By: Tilghman Lesher (tilghman) 2006-08-05 11:14:39

No, it appears the other 3 are unrelated.  I'm attaching a patch to update meetme to linkedlists, as it's getting to be impossible to debug without using the standard list macros.  This should also solve several possible races within meetme.
By: sedwards (sedwards) 2006-08-05 12:19:53

bt-8732 and bt-9967 are from crashes last night. I'll apply Corydon76's patch now.
By: sedwards (sedwards) 2006-08-05 12:35:33

Oops -- my major bad.

At the request of vechers I "upgraded" to SVN-branch-1.2-r38825 because of a SIP problem I'm also having -- bug 7664. Evidently, this breaks Corydon76's patch.

bt-8732 and bt-9967 are from SVN-branch-1.2-r38825.

gcc  -pipe  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3  -Iinclude -I../include -D_REENTRANT -D_GNU_SOURCE   -march=i686 -DZAPTEL_OPTIMIZATIONS         -fPIC   -c -o app_meetme.o app_meetme.c
app_meetme.c:139: warning: declaration does not declare anything
app_meetme.c:150:37: macro "AST_LIST_ENTRY" passed 2 arguments, but takes just 1
app_meetme.c:150: error: syntax error before "AST_LIST_ENTRY"
app_meetme.c:150: warning: no semicolon at end of struct or union
app_meetme.c:162:36: macro "AST_LIST_ENTRY" passed 2 arguments, but takes just 1
app_meetme.c:162: error: syntax error before "AST_LIST_ENTRY"
app_meetme.c:162: warning: no semicolon at end of struct or union
app_meetme.c:172: error: 'listen' redeclared as different kind of symbol
/usr/include/sys/socket.h:199: error: previous declaration of 'listen' was here
app_meetme.c:172: error: 'listen' redeclared as different kind of symbol
/usr/include/sys/socket.h:199: error: previous declaration of 'listen' was here
app_meetme.c:173: error: syntax error before '}' token
app_meetme.c: In function `set_talk_volume':
app_meetme.c:322: error: dereferencing pointer to incomplete type
By: Tilghman Lesher (tilghman) 2006-08-05 14:02:58

Oops, didn't realize my test hadn't compiled app_meetme, so I missed a few things.  Try this new patch.
By: sedwards (sedwards) 2006-08-07 12:13:40

Out of the frying pan and into the fire :)

The 20060805__bug7647__2.diff.txt patch causes Asterisk to crash every 2 minutes. core.14223-bt, core.13871-bt, core.17034-bt may help.
By: Serge Vecher (serge-v) 2006-08-31 13:05:14

ping
By: sedwards (sedwards) 2006-08-31 17:04:03

I've upgraded to 1.2.11 -- still crashing several times a day :(

I've also recoded my 2 applications (play a random file and write a custom CDR) as AGI's so there is no "custom" code involved with the asterisk executable.

I've uploaded a "bt full" from the most recent crash.
By: sedwards (sedwards) 2006-09-28 21:11:27

I've upgraded to 1.2.12.1. Still no joy -- 8 crashes in the last 24 hours.

Most of the "bt's" look like this:

#0  0x005e67a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#0  0x005e67a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x006267a5 in raise () from /lib/tls/libc.so.6
#2  0x00628209 in abort () from /lib/tls/libc.so.6
#3  0x0065a71a in __libc_message () from /lib/tls/libc.so.6
#4  0x00660fbf in _int_free () from /lib/tls/libc.so.6
ASTERISK-1  0x0066133a in free () from /lib/tls/libc.so.6
ASTERISK-2  0x08061416 in ast_channel_free (chan=0xb780b988) at channel.c:959
ASTERISK-3  0x080629fa in ast_hangup (chan=0xb780b988) at channel.c:1392
ASTERISK-4  0x001a94ec in conf_free (conf=0xb7827a30) at app_meetme.c:789
ASTERISK-5  0x001abf45 in conf_run (chan=0x91a9670, conf=0xb7827a30, confflags=4096, optargs=0xb7973d4c) at app_meetme.c:1602
ASTERISK-6 0x001adab0 in conf_exec (chan=0x91a9670, data=0xb7978070) at app_meetme.c:2024
ASTERISK-7 0x08083a5b in pbx_exec (c=0x91a9670, app=0x90cb808, data=0xb7978070, newstack=1) at pbx.c:553
ASTERISK-8 0x08086ea3 in pbx_extension_helper (c=0x91a9670, con=0x0, context=0x91a97c0 "meetme-menu", exten=0x91a98b4 "2",
   priority=3, label=0x0, callerid=0x914c128 "xxxxxxxxxx", action=1) at pbx.c:1697
ASTERISK-9 0x0808811d in ast_spawn_extension (c=0x91a9670, context=0x91a97c0 "meetme-menu", exten=0x91a98b4 "2", priority=3,
   callerid=0x914c128 "xxxxxxxxxx") at pbx.c:2230
ASTERISK-10 0x0808859a in __ast_pbx_run (c=0x91a9670) at pbx.c:2296
ASTERISK-11 0x080892ef in pbx_thread (data=0x91a9670) at pbx.c:2517
ASTERISK-12 0x007c6371 in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-13 0x006c6ffe in clone () from /lib/tls/libc.so.6
By: sedwards (sedwards) 2006-09-28 21:25:52

I've uploaded a "bt-full" from a recent crash.

Note: the one without the "-b" should be deleted. Unfortunately I don't have permission...
By: Serge Vecher (serge-v) 2006-09-29 12:07:40

sedwards: please always include the version/revision of Asterisk you are attaching the bt for ... Can you please check if this happens with the latest svn 1.2 branch code?
By: sedwards (sedwards) 2006-10-02 12:53:09

SVN-branch-1.2-r43977

Still no joy -- 12 crashes in the last 24 hours. My boss is losing confidence in Asterisk :(

Most of the bt's & bt-full's are exactly the same, just the pointers and the data changes.
By: Serge Vecher (serge-v) 2006-10-02 13:34:24

sedwards, by the way, are these bt's from Asterisk built with 'make dont-optimize'? If not, they will need to be redone ...
By: sedwards (sedwards) 2006-10-02 13:50:46

I use the MAKECMDGOALS approach. This is a snippet from my build script:

export MAKECMDGOALS=dont-optimize
cd /tmp/asterisk-1.2
make
sudo make install
By: sedwards (sedwards) 2006-10-06 16:24:49

SVN-branch-1.2-r43977

25 crashes in the last 4 days. Any clues would be greatly appreciated. Either I'm pushing harder on meetme that others or I'm doing something wrong.


By: Serge Vecher (serge-v) 2006-10-06 16:33:59

sedwards: is the latest code from 1.4 branch any better?
By: sedwards (sedwards) 2006-10-06 16:48:37

SVN-branch-1.2-r43977

Moving to 1.4 is not an option right now.
By: sedwards (sedwards) 2006-10-30 17:14:15.000-0600

1.2.13

I upgraded to 1.2.13 -- no joy, still averaging 7+ crashes a day.

I booted a non-SMP kernel and Asterisk has not crashed in 4 days!

IMNSHO, something is broken in meetme. 90%+ of the xxx's of crashes were in meetme -- either in freeing a channel or freeing a conference.

After almost 3 months of no interest from the bug marshals, I've also lost interest. Please close this bug.


By: Tilghman Lesher (tilghman) 2006-10-30 17:17:47.000-0600

It's not a matter of there being no interest.  It's a matter of the bug being incredibly difficult to track down.
By: sedwards (sedwards) 2006-10-30 17:31:08.000-0600

I don't mean to sound ungrateful, I was just feeling pretty lonely -- the last constructive input was from you back on 8/5/6.

I am kicking myself for not thinking of something as obvious as "try turning off SMP."

I know that's not as good as a patch, but it does indicate where the bug is not -- my code or the way I [ab]use meetme.

If someone has an idea or wants me to try something, please feel free to ask.
By: Peng Yong (ppyy) 2006-10-31 22:00:20.000-0600

asterisk 1.2.12.1, crash in app_meetme. bt uploaded.
By: Russell Bryant (russell) 2007-02-16 18:10:41.000-0600

I have uploaded a patch that may solve this issue for you.  It is essentially a backport of the reference counting that is done on conferences in 1.4 and trunk.  This patch solved random crashes for another user of MeetMe in 1.2.  However, this patch isn't perfect, as we have some complaints of some conferences not going away in 1.4, as well as 1.2 using this patch.  However, that is better than crashing.  :)
By: Russell Bryant (russell) 2007-02-19 13:31:13.000-0600

I have posted an updated patch that will hopefully solve the issue with conferences not going away, as well.
By: Russell Bryant (russell) 2007-02-20 18:19:24.000-0600

This should be fixed in 1.2 in revision 55750.

Thanks