Summary:ASTERISK-05900: Crash in meetme: *** glibc detected *** double free or corruption (!prev): (0x......)
Reporter:kuj (kuj)Labels:
Date Opened:2005-12-24 14:23:15.000-0600Date Closed:2008-01-15 16:08:54.000-0600
Versions:Frequency of
Environment:Attachments:( 0) bt.txt
( 1) bt3.txt
( 2) bt4.txt
( 3) bt5.txt
( 4) bt8.txt
( 5) console3.txt
( 6) debug.txt
( 7) debug3.txt
( 8) debug4.txt
( 9) debugfull5.txt
(10) debugfull8.txt
(11) extensions.conf
(12) sipcrash2.txt
Description:This may be another datapoint in the series of recently reported crashes.
Scenario: UA1 (Polycom) dials into conference line, is placed into conference as a user, waiting for admin, listening to MOH. UA2 (eyebeam) dialing into same conf line (Ext. 2600 in attached extensions.conf), entering conf room number as prompted by Background command, but intention is to enter as admin (11-digit room number per attached extension.conf). While entering conf room number, and before entering full 11-digit room no. or being placed into Meetme, * crashes according to attached bt.txt. Note that UA2 has not been placed into conference yet.

Attached files (bt, debug, sipcrash2 console log) were captured from * built with dont-optimize, so should yield valid backtrace. While it crashes in app_meetme, I doubt the root cause to this is in app_meetme, so feel free to reclassify.

This crash is reproducible here, although the exact number of DTMF digits required to crash * varies.


This crash does not happen on SVN trunk version 7520M or earlier.
Comments:By: Tilghman Lesher (tilghman) 2005-12-24 16:57:22.000-0600

Additional backtrace info needed:

(gdb) frame 6
(gdb) p *fr

By: Tilghman Lesher (tilghman) 2005-12-24 17:01:03.000-0600

Come to think of it, please apply the debugging patch is 6032.  Being able to see two different systems with the same bogus frame crash may help to diagnose the issue faster.

By: Mark Spencer (markster) 2005-12-24 18:28:33.000-0600

Should be fixed in SVN trunk ASTERISK-7424, sorry about that.  Feel free to reopen if the problem still occurs.

By: kuj (kuj) 2005-12-24 21:01:57.000-0600

Gotta reopen, as the latest meetme changes don't fix it. I'll attach more files, one set just with the meetme change (7620), the other built on 7620 with Corydon's debug patch from ASTERISK-5976032.

By: kuj (kuj) 2005-12-24 21:07:54.000-0600

Files uploaded. bt3, console3 and debug3 go together (SVN trunk 7620). bt4 and debug4 are the other set (SVN trunk 7620 + Corydon's debug patch). debug4 contains a "full" logger config, thus no separate console log.

By: Tilghman Lesher (tilghman) 2005-12-24 23:17:29.000-0600

Could you rerun the bt4 with error,warning,notice turned on, as well as debug,verbose (in logger.conf)?

By: kuj (kuj) 2005-12-25 13:31:09.000-0600

Here you go: debugfull5 has a "full" log as requested. I believe debug4 had that as well. However, I couldn't find your debug prints (from the patch in ASTERISK-5976032) in either of them. I did verify the patch was applied, though. Rebuilt from scratch also (make clean dont-optimize).

By: Tilghman Lesher (tilghman) 2005-12-25 14:18:30.000-0600

Uh, there's something wrong with your "full" log.  That only has verbose and debug enabled.  I'd expect to see at least a few NOTICE, WARNING, and ERROR messages.

Perhaps you forgot to do a 'logger reload' after changing logger.conf ?

By: kuj (kuj) 2005-12-25 14:36:08.000-0600

Nothing wrong with the log. I just trimmed it to a "relevant" time window: from when UA2 (eyebeam softphone) registers, to when the crash occurs. No ERRORs, WARNINGs or NOTICEs are logged during that time. I do see a few NOTICEs and WARNINGs prior to the softphone registering, no ERRORs though. None of those logs are from your debug code.

Right now I'm single-stepping back through the svn timeline, trying to determine which updates are causing this issue.

By: kuj (kuj) 2005-12-25 15:14:45.000-0600

svn trunk 7547M introduces the issue. Anything before that does not crash.
I can take the latest svn (7626) and just replace app_meetme.c with a pre-7547 version and it works fine.
However, I suspect that the changes to channel.[ch] in 7547 may also play a role, as the crash is triggered *before* UA2 is placed into the meetme. (Recall that UA1 is waiting in the meetme room, waiting with MoH for the leader to arrive. I then start dialing on UA2, which causes the crash to occur. On UA2, I have not yet been prompted for the conf. room no., though, so UA2 cannot be in the meetme yet)

By: Mark Spencer (markster) 2005-12-25 17:13:51.000-0600

I'm having some trouble with your backtrace.  Versions 7620 to 7626 have a "break" at line 2276, where your backtrace claims to have an ast_frfree()...  Are you using any features with meetme?  If so, if you use "plain meetme" does the problem still occur?

By: Mark Spencer (markster) 2005-12-25 17:40:02.000-0600

I am theorizing that the crash has to do with the fact that we're using just one pseudo channel for both making announcements and for recording the channel / doing the conversions.  I think we may have to break down and open *two* pseudos, one for announcements, one for recording, alas.

By: Mark Spencer (markster) 2005-12-25 17:47:50.000-0600

Okay, it should *really* be fixed no in SVN trunk 7627.  Again feel free to reopen if i still didnt' get it :)  Merry Christmas / Happy Holidays!

By: Russell Bryant (russell) 2005-12-25 18:06:48.000-0600

these crashes are related to optimizations only present in the trunk, so no changes are necessary for the 1.2 branch

By: kuj (kuj) 2005-12-25 18:27:39.000-0600

Don't know what to say, but the mystery goes on.
Same thing happens with 7627 applied. To be on the safe side, I got rid of existing sources, downloaded complete source (7627), deleted all existing modules, built source with dont-optimize, installed, had to add format_mp3.so as single "foreign" module (after compiling it against the 7627 source tree) and could still replicate the problem. See bt8 and debugfull8. This time, the line number from the backtrace matches the real source. Sorry for that, don't know how it happened.

Merry Christmas to you guys, too! I'm going to give in for the day now!

By: Mark Spencer (markster) 2005-12-26 16:21:59.000-0600

You've got me stumped.  Sounds like we're going to have to find a way to get together on IRC and go through this together.  Find me as "kram" in irc.freenode.net.  I will need root access on your machine and will need you to create the problem.  Thanks!

By: Mark Spencer (markster) 2005-12-26 16:56:56.000-0600

Okay also trying another fix attempt.  Update to latest SVN trunk ASTERISK-7442 and see if it goes away!

By: kuj (kuj) 2005-12-26 17:20:48.000-0600

Mark, 7640 was it! Thanks much!

7627 (local channel variation), when run together with 7640, yielded some really bad (i.e. scratchy, noisy) sound when users were placed into the conference after the conf admin joined (conf join announcement was bad, afterwards ok). I rolled back app_meetme.c to 7620 and ran it together with 7640 without those problems. No crash so far. I recommend to roll back app_meetme.c to 7620 and not use the local channel variation of 7627.

Thanks again!

By: Matt O'Gorman (mogorman) 2006-01-10 09:57:28.000-0600

Marko... close your bugs... fixed in commmit 7640

By: Digium Subversion (svnbot) 2008-01-15 16:08:48.000-0600

Repository: asterisk
Revision: 7620

U   trunk/apps/app_meetme.c

r7620 | markster | 2008-01-15 16:08:48 -0600 (Tue, 15 Jan 2008) | 2 lines

Fix multiple free of a frame (bug ASTERISK-5900)



By: Digium Subversion (svnbot) 2008-01-15 16:08:54.000-0600

Repository: asterisk
Revision: 7627

U   trunk/apps/app_meetme.c

r7627 | markster | 2008-01-15 16:08:54 -0600 (Tue, 15 Jan 2008) | 3 lines

Add "local channel" variation so that we don't read/write to the same
channel...  (bug ASTERISK-5900)