[Home]

Summary:ASTERISK-07045: Crash when using GROUP application.
Reporter:Jon Leren Schøpzinsky (jonleren)Labels:
Date Opened:2006-05-29 02:07:20Date Closed:2006-06-12 19:36:06
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:
Description:Ive seen this issue reported before, with the crash appearing when using GROUP applications under heavy load, but the issue went dead.
Attached is a full backtrace of the crash.

****** ADDITIONAL INFORMATION ******

#0  ast_var_name (var=0x20202020) at chanvars.c:71
       name = 0xffffffff <Address 0xffffffff out of bounds>
#1  0x080893b8 in pbx_builtin_getvar_helper (chan=0xb6ab40f8, name=0xb64ee460 "GROUP") at pbx.c:5935
       variables = (struct ast_var_t *) 0x20202020
       ret = 0x0
       i = 0
       places = warning: Unhandled dwarf expresion opcode DW_OP_piece
{0xb6ab44c0, 0x81222a4}
#2  0x080a1a54 in ast_app_group_get_count (group=0xb64ee5a0 "haslev-doctor-1-1", category=0xb64ee550 "GROUP") at app.c:1060
       chan = (struct ast_channel *) 0xb6ab40f8
       count = 0
       test = 0x0
       cat = "GROUP", '\0' <repeats 15 times>, "\030\000ð¶\000\000\000\000\020", '\0' <repeats 19 times>, "ôo\177\000\207\f\210\000\020\000ð¶\030]0·¬^0·(Á\233¶¬^0·Å¹\005\b"
#3  0xb73030d6 in group_count_exec (chan=0xb6f238e0, data=0x0) at app_groupcount.c:76
       count = 0
       u = (struct localuser *) 0xb69bc128
       group = "haslev-doctor-1-1", '\0' <repeats 62 times>
       category = "GROUP", '\0' <repeats 74 times>
       ret = '\0' <repeats 79 times>
       grp = Variable "grp" is not available.
Comments:By: Tilghman Lesher (tilghman) 2006-05-29 11:01:24

What the stack backtrace indicates is heap corruption.  It's going to take a bit more digging in gdb to figure out what is causing this.
By: Tilghman Lesher (tilghman) 2006-05-29 11:03:42

Find one of the developers on IRC, and be prepared to give remote root privileges to the system in question.  Do not delete the core file that caused this error, as it's going to be essential to uncovering what caused the heap corruption.
By: Andrey S Pankov (casper) 2006-05-29 11:14:12

Do you have some logs entries before the crash?
By: Jon Leren Schøpzinsky (jonleren) 2006-05-30 02:29:28

Heres the last "breath" of the log file.
Asterisk crashed at 08:00:18

I have altered the phone numbers, server address and ip addresses, as this is from a live production server.

May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '3634****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '9843****' is now UNREACHABLE!  Last qualify: 29
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '9720****' is now UNREACHABLE!  Last qualify: 55
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8827****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '3284****' is now UNREACHABLE!  Last qualify: 42
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 49
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8820****' is now TOO LAGGED! (3187ms / 3000ms)
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8827****' is now TOO LAGGED! (3189ms / 500ms)
May 29 08:00:14 NOTICE[19041] chan_sip.c: Peer '8827****' is now REACHABLE! (204ms / 500ms)
May 29 08:00:15 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 51
May 29 08:00:15 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 54
May 29 08:00:15 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 1418
May 29 08:00:15 NOTICE[19041] chan_sip.c: Peer '4517****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:15 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 214
May 29 08:00:15 ERROR[19041] chan_sip.c: Got SUBSCRIBE for extensions without hint. Please add hint to 4090**** in context bn_abo
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 262
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8730****' is now UNREACHABLE!  Last qualify: 60
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 52
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '3259****' is now UNREACHABLE!  Last qualify: 40
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 46
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 52
May 29 08:00:16 ERROR[19041] chan_sip.c: Got SUBSCRIBE for extensions without hint. Please add hint to 8838**** in context bn_abo
May 29 08:00:16 NOTICE[19041] chan_sip.c: Peer '8832****' is now TOO LAGGED! (3818ms / 3000ms)
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 51
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 61
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 71
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 62
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 62
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '7581****' is now UNREACHABLE!  Last qualify: 72
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 71
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 72
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 69
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 72
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now UNREACHABLE!  Last qualify: 81
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8838****' is now UNREACHABLE!  Last qualify: 1
May 29 08:00:18 NOTICE[19041] chan_sip.c: Registration from '<sip:0@voip1.voip.****>' failed for '83.221.139****' - Username/auth name mismatch
May 29 08:00:18 NOTICE[19041] chan_sip.c: Registration from '<sip:8820****@voip1.voip.****>' failed for '83.221.139****' - Wrong password
May 29 08:00:18 NOTICE[19041] chan_sip.c: Registration from '<sip:0@voip1.voip.****>' failed for '83.221.139****' - Username/auth name mismatch
May 29 08:00:18 NOTICE[19041] chan_sip.c: Registration from '<sip:0@voip1.voip.****>' failed for '83.221.139****' - Username/auth name mismatch
May 29 08:00:18 NOTICE[19041] chan_sip.c: Peer '8820****' is now REACHABLE! (1708ms / 3000ms)


It should be noted, that at the time of the crash, we had an mpg123 process, that had gone haywire, and the system load was about 20.0 due to this. Maybe that could have provoked a heap corruption.
By: Andrey S Pankov (casper) 2006-05-30 07:16:56

Is that reproducible at all or just a single "never happened before" crash?
By: Tilghman Lesher (tilghman) 2006-05-30 15:26:47

Heap corruption can only happen within the same process, not from another process.
By: Jon Leren Schøpzinsky (jonleren) 2006-05-30 16:16:23

This has only happended once. And it was under stress from many Zap channels, alof of sound playback and queuing. But still shouldnt cause heap corruption, as far as I know.
By: Serge Vecher (serge-v) 2006-06-05 16:10:10

jonleren: was this actually 1.2.7.1 or trunk? Is this reproducible in the latest 1.2.8 or trunk?
By: Jon Leren Schøpzinsky (jonleren) 2006-06-06 01:19:24

This was 1.2.7.1, but unforunatly we have been forced to downgrade all of our servers to 1.2.0, due to problems with sound quality on the IAX2 channel, in later versions of asterisk. I will open a new issue on this, as soon as we confirm that this is indeed a problem.
By: Serge Vecher (serge-v) 2006-06-12 19:36:06

jonleren: if are indeed able to reproduce this in the latest stable (1.2.9.1 ATM), please do not open a new bug, bug reopen this one. Also, please note that there was a major security issue resolved in 1.2.9.1