[Home]

Summary:ASTERISK-13226: Address out of bounds in queue_log using transfer
Reporter:Matt Riddell (zx81)Labels:
Date Opened:2008-12-15 20:31:53.000-0600Date Closed:2009-01-14 18:15:27.000-0600
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Applications/app_queue
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 14086.patch
( 1) 14086v2.patch
Description:This system has been up without problems for around 100 days until this week at which stage it has crashed twice:

#0  0xb7dcd463 in strlen () from /lib/tls/i686/cmov/libc.so.6
#1  0xb7da1164 in vfprintf () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7da62e2 in fprintf () from /lib/tls/i686/cmov/libc.so.6
#3  0x080aff57 in ast_queue_log (queuename=0x18 <Address 0x18 out of bounds>, callid=0xb7e8 <Address 0xb7e8 out of bounds>,
   agent=0x8cfd518 "SIP/8780", event=0xb749ffee "TRANSFER", fmt=0xb749ffe0 "%s|%s|%ld|%ld") at logger.c:359
#4  0xb7491933 in queue_transfer_fixup (data=0x8c9bf90, old_chan=0xb5fbb868, new_chan=0xb5f9fef0) at app_queue.c:2582
ASTERISK-1  0x0808428d in ast_do_masquerade (original=0xb5f9fef0) at channel.c:3537
ASTERISK-2  0x080867d9 in __ast_read (chan=0xb5f9fef0, dropaudio=0) at channel.c:1971
ASTERISK-3  0x08089822 in ast_channel_bridge (c0=0xb5f9fef0, c1=0xb5f9fef0, config=0xb6af8e7c, fo=0xb6af7f88, rc=0xb6af7f84)
   at channel.c:2366
ASTERISK-4  0xb7c5659d in ast_bridge_call (chan=0xb5f9fef0, peer=0x8d830c0, config=0xb6af8e7c) at res_features.c:1486
ASTERISK-5  0xb7b4d37d in dial_exec_full (chan=0xb5f9fef0, data=<value optimized out>, peerflags=0xb6af8f44, continue_exec=0x0)
   at app_dial.c:1775
ASTERISK-6 0xb7b4d7e2 in dial_exec (chan=0xb5f9fef0, data=0xb6afafb8) at app_dial.c:1829
ASTERISK-7 0x080cd947 in pbx_extension_helper (c=0xb5f9fef0, con=0x0, context=0xb5fa0070 "internal", exten=0xb5fa00c0 "10800226440",
   priority=1, label=0x0, callerid=0xb678e7d0 "8721", action=E_SPAWN) at pbx.c:537
ASTERISK-8 0x080cf931 in __ast_pbx_run (c=0xb5f9fef0) at pbx.c:2317
ASTERISK-9 0x080d098e in pbx_thread (data=0xb5f9fef0) at pbx.c:2621
ASTERISK-10 0x080ff5d0 in dummy_start (data=0xb64b1070) at utils.c:912
ASTERISK-11 0xb7f12240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-12 0xb7e2d49e in clone () from /lib/tls/i686/cmov/libc.so.6

and

#0  0xb7ddb43b in strlen () from /lib/tls/i686/cmov/libc.so.6
#1  0xb7daf164 in vfprintf () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7db42e2 in fprintf () from /lib/tls/i686/cmov/libc.so.6
#3  0x080aff57 in ast_queue_log (queuename=0x20c62e <Address 0x20c62e out of bounds>,
   callid=0x493ece85 <Address 0x493ece85 out of bounds>, agent=0xcd87eb0 "SIP/8846", event=0xb74e9fee "TRANSFER",
   fmt=0xb74e9fe0 "%s|%s|%ld|%ld") at logger.c:359
#4  0xb74db933 in queue_transfer_fixup (data=0xd2b21b0, old_chan=0xdc9d218, new_chan=0xdc73aa8) at app_queue.c:2582
ASTERISK-1  0x0808428d in ast_do_masquerade (original=0xdc73aa8) at channel.c:3537
ASTERISK-2  0x080867d9 in __ast_read (chan=0xdc73aa8, dropaudio=0) at channel.c:1971
ASTERISK-3  0x08089822 in ast_channel_bridge (c0=0xdc73aa8, c1=0xdc73aa8, config=0xb4efae7c, fo=0xb4ef9fa8, rc=0xb4ef9fa4)
   at channel.c:2366
ASTERISK-4  0xb7c6459d in ast_bridge_call (chan=0xdc73aa8, peer=0xdc79a50, config=0xb4efae7c) at res_features.c:1486
ASTERISK-5  0xb7b9737d in dial_exec_full (chan=0xdc73aa8, data=<value optimized out>, peerflags=0xb4efaf44, continue_exec=0x0)
   at app_dial.c:1775
ASTERISK-6 0xb7b977e2 in dial_exec (chan=0xdc73aa8, data=0xb4efcfb8) at app_dial.c:1829
ASTERISK-7 0x080cd947 in pbx_extension_helper (c=0xdc73aa8, con=0x0, context=0xdc73c28 "internal", exten=0xdc73c78 "5765", priority=1,
   label=0x0, callerid=0xdc00e90 "8897", action=E_SPAWN) at pbx.c:537
ASTERISK-8 0x080cf931 in __ast_pbx_run (c=0xdc73aa8) at pbx.c:2317
ASTERISK-9 0x080d098e in pbx_thread (data=0xdc73aa8) at pbx.c:2621
ASTERISK-10 0x080ff5d0 in dummy_start (data=0x91d8e50) at utils.c:912
ASTERISK-11 0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-12 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6
Comments:By: Matt Riddell (zx81) 2008-12-15 20:35:54.000-0600

Hmmm, I wonder if this is related to 13047
By: Matt Riddell (zx81) 2008-12-15 20:39:00.000-0600

Thing is, we're using AddQueueMember etc with SIP/xxx, not local dialplan
By: Matt Riddell (zx81) 2008-12-15 20:48:25.000-0600

The revision we're running is actually since that patch
By: Matt Riddell (zx81) 2008-12-16 06:38:50.000-0600

Any ideas as to where I might look or information I could provide?
By: Mark Michelson (mmichelson) 2008-12-16 10:06:38.000-0600

Is this related to issue ASTERISK-13204? I'm curious to know if the datastore handling that was fixed there was the same underlying issue for this crash.
By: Matt Riddell (zx81) 2008-12-16 14:47:45.000-0600

There's no mention of the datastore in that backtrace, should I apply the patch anyway and see what happens?
By: Matt Riddell (zx81) 2008-12-16 14:50:32.000-0600

I tried applying the patch but it failed with lots of chunks.  Have updated the system to latest 1.4 svn - will need to wait for some down time to restart - unless it crashes :)
By: Matt Riddell (zx81) 2008-12-18 12:02:44.000-0600

Ok, I've restarted the system with latest 1.4 svn.  Am about 4 hours from their opening hours.  Will update once I see how things go.
By: Amilcar S Silvestre (amilcar) 2008-12-28 19:25:44.000-0600

Any news, ZX81?
By: Matt Riddell (zx81) 2009-01-11 18:19:12.000-0600

Still crashing.

Will upload cores.
By: Matt Riddell (zx81) 2009-01-11 18:24:43.000-0600

bt says:

#0  0x08080158 in ast_channel_datastore_find (chan=0xb681b360, info=0xb74bb104, uid=0x0) at channel.c:1409
#1  0xb74b0e60 in try_calling (qe=0xb6c88d90, options=<value optimized out>, announceoverride=0x0, url=0x0, tries=0xb6c88f30,
   noption=0xb6c88f2c, agi=0x0) at app_queue.c:2636
#2  0xb74b5058 in queue_exec (chan=0xb681b360, data=0xb6c8afb8) at app_queue.c:3999
#3  0x080cd2d7 in pbx_extension_helper (c=0xb681b360, con=0x0, context=0xb681b4e0 "internal", exten=0xb681b530 "379", priority=1,
   label=0x0, callerid=0xb6a7f8b0 "àË£¶@ù§¶\020", action=E_SPAWN) at pbx.c:537
#4  0x080cf7c1 in __ast_pbx_run (c=0xb681b360) at pbx.c:2318
ASTERISK-1  0x080d081e in pbx_thread (data=0xb681b360) at pbx.c:2622
ASTERISK-2  0x08100800 in dummy_start (data=0xb68200a8) at utils.c:850
ASTERISK-3  0xb7ef2240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e0d49e in clone () from /lib/tls/i686/cmov/libc.so.6
By: Matt Riddell (zx81) 2009-01-11 19:51:05.000-0600

Version is Asterisk SVN-branch-1.4-r164806
By: Matt Riddell (zx81) 2009-01-11 20:00:19.000-0600

Channel.c 1409 is:

AST_LIST_TRAVERSE_SAFE_BEGIN(&chan->datastores, datastore, entry) {

app_queue.c 2636 is:

return ast_channel_datastore_find(chan, &queue_transfer_info, NULL) ? 0 : 1;

app_queue.c 3999 is:

/* Try calling all queue members for 'timeout' seconds */
res = try_calling(&qe, args.options, args.announceoverride, args.url, &tries, &noption, args.agi);

pbx.c 537 is:

res = app->execute(c, S_OR(data, ""));

pbx.c 2318 is:

return pbx_extension_helper(c, NULL, context, exten, priority, NULL, callerid, E_SPAWN);
By: Leif Madsen (lmadsen) 2009-01-11 22:03:22.000-0600

Whenever I see "options=<value optimized out>" in a backtrace, from what I can tell that means you need to enable DONT_OPTIMIZE in the compiler flags options.
By: Matt Riddell (zx81) 2009-01-12 14:38:30.000-0600

Can do, just means we need to wait for another system crash of a production system.
By: Matt Riddell (zx81) 2009-01-12 14:40:52.000-0600

Ok recompiled, make installed, just need to wait for another crash or the end of the day before I restart it.  

If anyone can see anything in the meantime it would be great as this means I can test a change without waiting two days.
By: Mark Michelson (mmichelson) 2009-01-12 15:40:59.000-0600

This looks like a case of a datastore being freed but not removed from the datastore list on the channel. I'll see if I can find how this might be happening. I'm assuming the badly behaving datastore is the queue transfer datastore, but I can't really be certain just from what's provided.
By: Mark Michelson (mmichelson) 2009-01-12 15:53:38.000-0600

I have found a potential trouble spot with the queue transfer datastore. I'll upload a patch in just a sec for testing.
By: Mark Michelson (mmichelson) 2009-01-12 15:58:28.000-0600

I found that the queue_transfer_fixup was searching for and removing the queue transfer datastore from the wrong channel. It's hard for me to picture the necessary circumstances for this problem to actually occur, but the code is clearly wrong, and it's the only thing that I see that seems likely to be causing the crash.

Patch 14086.patch corrects this behavior and may fix your crash too. Please let me know if it helps.
By: Matt Riddell (zx81) 2009-01-12 17:10:57.000-0600

Heh, just crashed before applying, so I'll have to wait till tonight to test - will do the make install stuff and get back to you
By: Matt Riddell (zx81) 2009-01-12 17:13:09.000-0600

Ok, applied the patch, (DONT_OPTIMIZE is on now too), so I'll wait for a crash or tonight to restart - the files are in place, so it'll start up with the new version if it does crash.
By: Matt Riddell (zx81) 2009-01-12 19:32:30.000-0600

ok, it's just crashed again, so that patch is live
By: Matt Riddell (zx81) 2009-01-12 20:02:12.000-0600

didn't work - uploading core file
By: Matt Riddell (zx81) 2009-01-12 20:05:56.000-0600

#0  0x0808157c in ast_channel_datastore_free (datastore=0x8290880) at channel.c:1341
       res = 0
#1  0x08081122 in ast_channel_free (chan=0xb6b18218) at channel.c:1243
       fd = 0
       vardata = (struct ast_var_t *) 0x0
       f = (struct ast_frame *) 0x0
       headp = (struct varshead *) 0xb6b1855c
       datastore = (struct ast_datastore *) 0x8290880
       name = "ü1Ú¶Ðô±¶\000\000\000\000\000\000\000\000Ðô±¶8X%\b\230\203±¶¸}%\b\000\017}\000\2301Ú¶ü1Ú¶call,all", '\0' <repeats 27 times>
       dashptr = 0x0
       __PRETTY_FUNCTION__ = "ast_channel_free"
#2  0x08081ec6 in ast_hangup (chan=0xb6b18218) at channel.c:1553
       res = 0
       __PRETTY_FUNCTION__ = "ast_hangup"
#3  0x080c30be in __ast_pbx_run (c=0xb6b18218) at pbx.c:2562
       found = 1
       res = -1
       autoloopflag = 0
       error = 1
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
#4  0x080c328a in pbx_thread (data=0xb6b18218) at pbx.c:2622
       c = (struct ast_channel *) 0xb6b18218
ASTERISK-1  0x08100ac9 in dummy_start (data=0xb6b186c0) at utils.c:850
       _buffer = {__routine = 0x80698c3 <ast_unregister_thread>, __arg = 0xb6da3bb0, __canceltype = -1209005570, __prev = 0x0}
       ret = (void *) 0xb6da3468
       a = {start_routine = 0x80c3273 <pbx_thread>, data = 0xb6b18218,
 name = 0xb6b08c60 "pbx_thread", ' ' <repeats 11 times>, "started at [ 2646] pbx.c ast_pbx_start()"}
ASTERISK-2  0xb7fa5240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
ASTERISK-3  0xb7ec049e in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.
By: Matt Riddell (zx81) 2009-01-12 20:32:19.000-0600

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E4600  @ 2.40GHz
stepping        : 13
cpu MHz         : 2394.068
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 cx16 xtpr lahf_lm
bogomips        : 4791.72

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E4600  @ 2.40GHz
stepping        : 13
cpu MHz         : 2394.068
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl est tm2 cx16 xtpr lahf_lm
bogomips        : 4788.32

cat /proc/version
Linux version 2.6.18-5-686 (Debian 2.6.18.dfsg.1-13etch6) (dannf@debian.org) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Dec 18 21:24:20 UTC 2007

cat /proc/meminfo
MemTotal:      1035548 kB
MemFree:        128456 kB
Buffers:        139880 kB
Cached:         422272 kB
SwapCached:          0 kB
Active:         612872 kB
Inactive:       149904 kB
HighTotal:      129964 kB
HighFree:          236 kB
LowTotal:       905584 kB
LowFree:        128220 kB
SwapTotal:    39102136 kB
SwapFree:     39102080 kB
Dirty:            1316 kB
Writeback:           0 kB
AnonPages:      200620 kB
Mapped:          23104 kB
Slab:           113428 kB
PageTables:       1364 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  39619908 kB
Committed_AS:   410292 kB
VmallocTotal:   114680 kB
VmallocUsed:     24136 kB
VmallocChunk:    89528 kB

cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda2[0] sdb2[1]
     39102144 blocks [2/2] [UU]
     
md0 : active raid1 sda1[0] sdb1[1]
     117185984 blocks [2/2] [UU]
     
unused devices: <none>


By: Martin Vit (festr) 2009-01-13 05:35:30.000-0600

Hi

I'm seeing the same crashes with the latest 1.4 svn (i've a slight modified app_queue and channel.c but it crashes with vanilla asterisk too).

edit: and it seems that it crashes when accessing datastor->info structure.


#0  0x08088eb4 in ast_do_masquerade (original=0x8259570) at channel.c:3710
3710                            if (ds->info->chan_fixup)
(gdb) bt
#0  0x08088eb4 in ast_do_masquerade (original=0x8259570) at channel.c:3710
#1  0x08083987 in __ast_read (chan=0x8259570, dropaudio=0) at channel.c:2040
#2  0x080850f6 in ast_read (chan=0x8259570) at channel.c:2434
#3  0xb76df1fc in dial_exec_full (chan=0x0, data=0xb518fdfc, peerflags=0x8259570, continue_exec=0x82bdd50) at app_dial.c:514
#4  0xb76e2af6 in dial_exec_full (chan=0xb7f0ac64, data=0x8291e28, peerflags=0x8259570, continue_exec=0xb5188180) at utils.h:423


another crash

#0  0x08081c36 in ast_channel_datastore_free (datastore=0x8263cb8) at channel.c:1337
1337            if (datastore->info->destroy != NULL && datastore->data != NULL) {
(gdb) bt
#0  0x08081c36 in ast_channel_datastore_free (datastore=0x8263cb8) at channel.c:1337
#1  0x080817f7 in ast_channel_free (chan=0x8333e50) at channel.c:1239
#2  0x080825a6 in ast_hangup (chan=0x8333e50) at channel.c:1549
#3  0x080c3602 in __ast_pbx_run (c=0x8333e50) at pbx.c:2562
#4  0x080c37bc in pbx_thread (data=0x8333e50) at pbx.c:2622
ASTERISK-1  0x081006fa in dummy_start (data=0x834fe00) at utils.c:856
ASTERISK-2  0xb7fb6b63 in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-3  0xb7ee218a in clone () from /lib/tls/libc.so.6


yet another crash

#0  0xb7ddbe84 in _int_free () from /lib/tls/libc.so.6
(gdb) bt
#0  0xb7ddbe84 in _int_free () from /lib/tls/libc.so.6
#1  0xb7ddadcb in free () from /lib/tls/libc.so.6
#2  0xb75c5523 in try_calling (qe=0xb50ca848, options=0xffffffdc <Address 0xffffffdc out of bounds>, announceoverride=0x8289560 "ut", url=0x0, tries=0xb50cc8b8,
   noption=0x80817f7, agi=0x82832f8 "X??`\225(\b?H]?") at utils.h:360
#3  0x08081c59 in ast_channel_datastore_free (datastore=0x82832f8) at channel.c:1338
#4  0x080817f7 in ast_channel_free (chan=0x82db3c8) at channel.c:1239
ASTERISK-1  0x080825a6 in ast_hangup (chan=0x82db3c8) at channel.c:1549
ASTERISK-2  0x080c3602 in __ast_pbx_run (c=0x82db3c8) at pbx.c:2562
ASTERISK-3  0x080c37bc in pbx_thread (data=0x82db3c8) at pbx.c:2622
ASTERISK-4  0x081006fa in dummy_start (data=0x83bd748) at utils.c:856
ASTERISK-5  0xb7f15b63 in start_thread () from /lib/tls/libpthread.so.0
ASTERISK-6 0xb7e4118a in clone () from /lib/tls/libc.so.6


By: Mark Michelson (mmichelson) 2009-01-13 10:07:51.000-0600

All right. Let's synchronize here and make sure that this is all stemming from the same bad handling.

First of all, to ZX81, I hadn't realized when you said you were uploading the core files that you were literally doing that. I thought you were using the term "core" to mean "backtrace." The actual core files are not useful on a machine other than the one that crashed. Uploading backtrace output (as a file upload) would be more useful.

festr: All of those places where the crashes are occurring have to do with datastores, and with the exception of the last one are also indicative that we are accessing a datastore which has been freed but not removed from the channel's list of datastores. The last one looks like a double free of a datastore.

I'll take another look at the datastore handling in app_queue to see if I can see what's going wrong.
By: Matt Riddell (zx81) 2009-01-13 10:50:10.000-0600

bt:

#0  0x0808157c in ast_channel_datastore_free (datastore=0x82ca388) at channel.c:1341
#1  0x08081122 in ast_channel_free (chan=0x82e6378) at channel.c:1243
#2  0x08081ec6 in ast_hangup (chan=0x82e6378) at channel.c:1553
#3  0x080c30be in __ast_pbx_run (c=0x82e6378) at pbx.c:2562
#4  0x080c328a in pbx_thread (data=0x82e6378) at pbx.c:2622
ASTERISK-1  0x08100ac9 in dummy_start (data=0x8296660) at utils.c:850
ASTERISK-2  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-3  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

bt full:

#0  0x0808157c in ast_channel_datastore_free (datastore=0x82ca388) at channel.c:1341
       res = 0
#1  0x08081122 in ast_channel_free (chan=0x82e6378) at channel.c:1243
       fd = 0
       vardata = (struct ast_var_t *) 0x0
       f = (struct ast_frame *) 0x0
       headp = (struct varshead *) 0x82e66bc
       datastore = (struct ast_datastore *) 0x82ca388
       name = "ü±¤¶", '\0' <repeats 20 times>, "ød.\b`k.\b\000\017}\000\230±¤¶ü±¤¶call,all", '\0' <repeats 27 times>
       dashptr = 0x0
       __PRETTY_FUNCTION__ = "ast_channel_free"
#2  0x08081ec6 in ast_hangup (chan=0x82e6378) at channel.c:1553
       res = 0
       __PRETTY_FUNCTION__ = "ast_hangup"
#3  0x080c30be in __ast_pbx_run (c=0x82e6378) at pbx.c:2562
       found = 1
       res = -1
       autoloopflag = 0
       error = 1
       __PRETTY_FUNCTION__ = "__ast_pbx_run"
#4  0x080c328a in pbx_thread (data=0x82e6378) at pbx.c:2622
       c = (struct ast_channel *) 0x82e6378
ASTERISK-1  0x08100ac9 in dummy_start (data=0x8296660) at utils.c:850
       _buffer = {__routine = 0x80698c3 <ast_unregister_thread>, __arg = 0xb6a4bbb0, __canceltype = 0, __prev = 0x0}
       ret = (void *) 0x0
       a = {start_routine = 0x80c3273 <pbx_thread>, data = 0x82e6378,
 name = 0x82c29d8 "pbx_thread", ' ' <repeats 11 times>, "started at [ 2646] pbx.c ast_pbx_start()"}
ASTERISK-2  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
ASTERISK-3  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.
By: Matt Riddell (zx81) 2009-01-13 10:51:54.000-0600

Asterisk SVN-branch-1.4-r164806M built by root @ xxx.venturevoip.com on a i686 running Linux on 2009-01-13 02:34:44 UTC
By: Matt Riddell (zx81) 2009-01-13 10:57:12.000-0600

(gdb) thread apply all bt

Thread 40 (process 26850):
#0  0xb7f3c410 in ?? ()
#1  0xbfec2a88 in ?? ()
#2  0x00000001 in ?? ()
#3  0xbfec2bdb in ?? ()
#4  0xb7f2582b in __read_nocancel () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-1  0x0810a0c1 in read_char (el=0x817d1d8, cp=0xbfec2bdb "\b") at read.c:298
ASTERISK-2  0x08105232 in el_getc (el=0x817d1d8, cp=0xbfec2bdb "\b") at read.c:350
ASTERISK-3  0x0810563b in el_gets (el=0x817d1d8, nread=0xbfec2fd4) at read.c:243
ASTERISK-4  0x08071ed9 in main (argc=4, argv=0xbfec3234) at asterisk.c:3212
ASTERISK-5  0xb7d84ea8 in __libc_start_main () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-6 0x08058131 in ?? () at ../sysdeps/i386/elf/start.S:119

Thread 39 (process 26855):
#0  0xb7f3c410 in ?? ()
#1  0xb7d6e338 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x0806b577 in listener (unused=0x0) at asterisk.c:1000
ASTERISK-2  0x08100ac9 in dummy_start (data=0x817f1c0) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 38 (process 26856):
#0  0xb7f3c410 in ?? ()
#1  0xb7d32408 in ?? ()
#2  0xb7f29ff4 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#3  0x00000000 in ?? ()

Thread 37 (process 26857):
#0  0xb7f3c410 in ?? ()
#1  0xb7cf63f8 in ?? ()
#2  0x000001cd in ?? ()
#3  0x00000000 in ?? ()

Thread 36 (process 26858):
#0  0xb7f3c410 in ?? ()
#1  0xb7c990c8 in ?? ()
#2  0x00000000 in ?? ()

Thread 35 (process 26859):
#0  0xb7f3c410 in ?? ()
#1  0xb7b9c3a8 in ?? ()
#2  0x000003e8 in ?? ()
---Type <return> to continue, or q <return> to quit---
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080ac6ca in ast_io_wait (ioc=0x8187400, howlong=1000) at io.c:266
ASTERISK-2  0xb7bbe056 in do_monitor (data=0x0) at chan_mgcp.c:3514
ASTERISK-3  0x08100ac9 in dummy_start (data=0x819cc78) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 34 (process 26860):
#0  0xb7f3c410 in ?? ()
#1  0xb7aa5f38 in ?? ()
#2  0x0000270f in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb7b2d25b in pri_dchannel (vpri=0xb7b476e0) at chan_dahdi.c:8686
ASTERISK-2  0x08100ac9 in dummy_start (data=0x819e180) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 33 (process 26861):
#0  0xb7f3c410 in ?? ()
#1  0xb7a69f98 in ?? ()
#2  0x000003e8 in ?? ()
#3  0x00000017 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb7b26b18 in do_monitor (data=0x0) at chan_dahdi.c:6999
ASTERISK-2  0x08100ac9 in dummy_start (data=0x819e240) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 32 (process 26862):
#0  0xb7f3c410 in ?? ()
#1  0xb7a29418 in ?? ()
#2  0xb7a2e108 in ?? () from /usr/lib/asterisk/modules/pbx_spool.so
#3  0x00000000 in ?? ()

Thread 31 (process 26863):
#0  0xb7f3c410 in ?? ()
#1  0xb79cb298 in ?? ()
#2  0x00000000 in ?? ()

Thread 30 (process 26864):
#0  0xb7f3c410 in ?? ()
#1  0xb77173a8 in ?? ()
#2  0x000003e8 in ?? ()
#3  0x00000001 in ?? ()
---Type <return> to continue, or q <return> to quit---
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080ac6ca in ast_io_wait (ioc=0x81ca7d8, howlong=1000) at io.c:266
ASTERISK-2  0xb7722f28 in network_thread (ignore=0x0) at pbx_dundi.c:2161
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81c7078) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 29 (process 26865):
#0  0xb7f3c410 in ?? ()
#1  0xb76db1e8 in ?? ()
#2  0xb7e9cff4 in ?? () from /lib/tls/i686/cmov/libc.so.6
#3  0xb76db1d4 in ?? ()
#4  0xb7dfdab6 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb7dfd8db in sleep () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-2  0xb7723217 in process_precache (ign=0x0) at pbx_dundi.c:2239
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81c7208) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 28 (process 26866):
#0  0xb7f3c410 in ?? ()
#1  0xb769f3c8 in ?? ()
#2  0xb7e9cff4 in ?? () from /lib/tls/i686/cmov/libc.so.6
#3  0xb769f3b4 in ?? ()
#4  0xb7dfdab6 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb7dfd8db in sleep () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-2  0xb77230bb in process_clearcache (ignore=0x0) at pbx_dundi.c:2202
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81cbdc0) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 27 (process 26867):
#0  0xb7f3c410 in ?? ()
#1  0xb75d3358 in ?? ()
#2  0x00000001 in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080ac6ca in ast_io_wait (ioc=0x81e44c0, howlong=1) at io.c:266
ASTERISK-2  0xb761e67c in do_monitor (data=0x0) at chan_sip.c:16222
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81e90b0) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 26 (process 26868):
#0  0xb7f3c410 in ?? ()
#1  0xb757e2b8 in ?? ()
---Type <return> to continue, or q <return> to quit---
#2  0xb757e2ec in ?? ()
#3  0xb757e36c in ?? ()
#4  0xb7e34131 in select () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb75823e9 in ast_select (nfds=43, rfds=0xb757e36c, wfds=0xb757e2ec, efds=0x0, tvp=0x0)
   at /usr/src/asterisk/include/asterisk/channel.h:1349
ASTERISK-2  0xb7582159 in sound_thread (arg=0x81f49d8) at chan_oss.c:629
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81f9bc0) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 25 (process 26869):
#0  0xb7f3c410 in ?? ()
#1  0xb75422b8 in ?? ()
#2  0xb75422ec in ?? ()
#3  0xb754236c in ?? ()
#4  0xb7e34131 in select () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0xb75823e9 in ast_select (nfds=45, rfds=0xb754236c, wfds=0xb75422ec, efds=0x0, tvp=0x0)
   at /usr/src/asterisk/include/asterisk/channel.h:1349
ASTERISK-2  0xb7582159 in sound_thread (arg=0x81f5090) at chan_oss.c:629
ASTERISK-3  0x08100ac9 in dummy_start (data=0x81e8c38) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 24 (process 26870):
#0  0xb7f3c410 in ?? ()
#1  0xb74de3e8 in ?? ()
#2  0x000001c9 in ?? ()
#3  0x00000000 in ?? ()

Thread 23 (process 26871):
#0  0xb7f3c410 in ?? ()
#1  0xb7420418 in ?? ()
#2  0xb74326c8 in ?? () from /usr/lib/asterisk/modules/chan_skinny.so
#3  0xb74203a0 in ?? ()
#4  0xb7f25a48 in accept () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-1  0xb742d47e in accept_thread (ignore=0x0) at chan_skinny.c:4585
ASTERISK-2  0x08100ac9 in dummy_start (data=0x82455b0) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 22 (process 26872):
#0  0xb7f3c410 in ?? ()
#1  0xb73e43a8 in ?? ()
#2  0x000003e8 in ?? ()
#3  0x00000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 21 (process 26873):
#0  0xb7f3c410 in ?? ()
#1  0xb71fc358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 20 (process 26874):
#0  0xb7f3c410 in ?? ()
#1  0xb71c0358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 19 (process 26875):
#0  0xb7f3c410 in ?? ()
#1  0xb7184358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 18 (process 26876):
#0  0xb7f3c410 in ?? ()
#1  0xb7148358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 17 (process 26877):
#0  0xb7f3c410 in ?? ()
#1  0xb710c358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 16 (process 26878):
#0  0xb7f3c410 in ?? ()
#1  0xb70d0358 in ?? ()
#2  0x0000002f in ?? ()
#3  0x00000000 in ?? ()

Thread 15 (process 26879):
#0  0xb7f3c410 in ?? ()
#1  0xb7094358 in ?? ()
#2  0x0000002d in ?? ()
#3  0x00000000 in ?? ()

Thread 14 (process 26880):
#0  0xb7f3c410 in ?? ()
#1  0xb7058358 in ?? ()
#2  0x0000002d in ?? ()
---Type <return> to continue, or q <return> to quit---
#3  0x00000000 in ?? ()

Thread 13 (process 26881):
#0  0xb7f3c410 in ?? ()
#1  0xb701c358 in ?? ()
#2  0x0000002d in ?? ()
#3  0x00000000 in ?? ()

Thread 12 (process 26882):
#0  0xb7f3c410 in ?? ()
#1  0xb6fe0358 in ?? ()
#2  0x0000002d in ?? ()
#3  0x00000000 in ?? ()

Thread 11 (process 26883):
#0  0xb7f3c410 in ?? ()
#1  0xb6fa43f0 in ?? ()
#2  0x0000065b in ?? ()
#3  0x00000000 in ?? ()

Thread 10 (process 26884):
#0  0xb7f3c410 in ?? ()
#1  0xb6f68388 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000002 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080ac6ca in ast_io_wait (ioc=0x8247930, howlong=-1) at io.c:266
ASTERISK-2  0xb7269998 in network_thread (ignore=0x0) at chan_iax2.c:9236
ASTERISK-3  0x08100ac9 in dummy_start (data=0x8254568) at utils.c:850
ASTERISK-4  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-5  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 9 (process 26885):
#0  0xb7f3c410 in ?? ()
#1  0xb6e4a368 in ?? ()
#2  0x00001388 in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080b9c44 in accept_thread (ignore=0x0) at manager.c:2419
ASTERISK-2  0x08100ac9 in dummy_start (data=0x82748c0) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 8 (process 26886):
#0  0xb7f3c410 in ?? ()
#1  0xb6e0e3e8 in ?? ()
---Type <return> to continue, or q <return> to quit---
#2  0xffffffff in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080702dc in monitor_sig_flags (unused=0x0) at asterisk.c:2661
ASTERISK-2  0x08100ac9 in dummy_start (data=0x8276380) at utils.c:850
ASTERISK-3  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-4  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 7 (process 26893):
#0  0xb7f3c410 in ?? ()
#1  0xb6d574a8 in ?? ()
#2  0x00000297 in ?? ()
#3  0x00000000 in ?? ()

Thread 6 (process 26894):
#0  0xb7f3c410 in ?? ()
#1  0xb6dd1d28 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000001 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080b9569 in get_input (s=0x8254f80, output=0xb6dd1db4 "") at manager.c:2275
ASTERISK-2  0x080b9749 in do_message (s=0x8254f80) at manager.c:2312
ASTERISK-3  0x080b9909 in session_do (data=0x8254f80) at manager.c:2337
ASTERISK-4  0x08100ac9 in dummy_start (data=0x825a158) at utils.c:850
ASTERISK-5  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-6 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 5 (process 27600):
#0  0xb7f3c410 in ?? ()
#1  0xb6af9858 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000002 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080827e9 in ast_waitfor_nandfds (c=0xb6af9ad4, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0xb6af9ad0)
   at channel.c:1760
ASTERISK-2  0x08082b19 in ast_waitfor_n (c=0xb6af9ad4, n=2, ms=0xb6af9ad0) at channel.c:1822
ASTERISK-3  0x080893be in ast_generic_bridge (c0=0x82c9f90, c1=0x829f720, config=0xb6afa27c, fo=0xb6afa000, rc=0xb6af9ffc, bridge_end=
     {tv_sec = 0, tv_usec = 0}) at channel.c:3948
ASTERISK-4  0x0808a8c0 in ast_channel_bridge (c0=0x82c9f90, c1=0x829f720, config=0xb6afa27c, fo=0xb6afa000, rc=0xb6af9ffc)
   at channel.c:4286
ASTERISK-5  0xb7ca1bbb in ast_bridge_call (chan=0x82c9f90, peer=0x829f720, config=0xb6afa27c) at res_features.c:1570
ASTERISK-6 0xb7bd8376 in dial_exec_full (chan=0x82c9f90, data=0xb6afd008, peerflags=0xb6afaed4, continue_exec=0x0) at app_dial.c:1835
ASTERISK-7 0xb7bd85c2 in dial_exec (chan=0x82c9f90, data=0xb6afd008) at app_dial.c:1882
ASTERISK-8 0x080bdbe8 in pbx_exec (c=0x82c9f90, app=0x819ae70, data=0xb6afd008) at pbx.c:537
ASTERISK-9 0x080c1000 in pbx_extension_helper (c=0x82c9f90, con=0x0, context=0x82ca110 "internal", exten=0x82ca160 "10800750750",
   priority=1, label=0x0, callerid=0x82563d8 "393", action=E_SPAWN) at pbx.c:1863
---Type <return> to continue, or q <return> to quit---
ASTERISK-10 0x080c20b2 in ast_spawn_extension (c=0x82c9f90, context=0x82ca110 "internal", exten=0x82ca160 "10800750750", priority=1,
   callerid=0x82563d8 "393") at pbx.c:2318
ASTERISK-11 0x080c24b8 in __ast_pbx_run (c=0x82c9f90) at pbx.c:2407
ASTERISK-12 0x080c328a in pbx_thread (data=0x82c9f90) at pbx.c:2622
ASTERISK-13 0x08100ac9 in dummy_start (data=0x8257360) at utils.c:850
ASTERISK-14 0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-15 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 4 (process 27873):
#0  0xb7f3c410 in ?? ()
#1  0xb6c4b788 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000004 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080827e9 in ast_waitfor_nandfds (c=0xb6c4ba04, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0xb6c4ba00)
   at channel.c:1760
ASTERISK-2  0x08082b19 in ast_waitfor_n (c=0xb6c4ba04, n=2, ms=0xb6c4ba00) at channel.c:1822
ASTERISK-3  0x080893be in ast_generic_bridge (c0=0x82557f8, c1=0x82c2a50, config=0xb6c4c19c, fo=0xb6c4bf30, rc=0xb6c4bf2c, bridge_end=
     {tv_sec = 0, tv_usec = 0}) at channel.c:3948
ASTERISK-4  0x0808a8c0 in ast_channel_bridge (c0=0x82557f8, c1=0x82c2a50, config=0xb6c4c19c, fo=0xb6c4bf30, rc=0xb6c4bf2c)
   at channel.c:4286
ASTERISK-5  0xb7ca1bbb in ast_bridge_call (chan=0x82557f8, peer=0x82c2a50, config=0xb6c4c19c) at res_features.c:1570
ASTERISK-6 0xb7bd8376 in dial_exec_full (chan=0x82557f8, data=0xb6c4ef28, peerflags=0xb6c4cdf4, continue_exec=0x0) at app_dial.c:1835
ASTERISK-7 0xb7bd85c2 in dial_exec (chan=0x82557f8, data=0xb6c4ef28) at app_dial.c:1882
ASTERISK-8 0x080bdbe8 in pbx_exec (c=0x82557f8, app=0x819ae70, data=0xb6c4ef28) at pbx.c:537
ASTERISK-9 0x080c1000 in pbx_extension_helper (c=0x82557f8, con=0x0, context=0x8255978 "macro-vis-incoming", exten=0x82559c8 "s",
   priority=3, label=0x0, callerid=0x82fead8 "021585404", action=E_SPAWN) at pbx.c:1863
ASTERISK-10 0x080c20b2 in ast_spawn_extension (c=0x82557f8, context=0x8255978 "macro-vis-incoming", exten=0x82559c8 "s", priority=3,
   callerid=0x82fead8 "021585404") at pbx.c:2318
ASTERISK-11 0xb6f091d6 in _macro_exec (chan=0x82557f8, data=0xb6c54008, exclusive=0) at app_macro.c:308
ASTERISK-12 0xb6f09eef in macro_exec (chan=0x82557f8, data=0xb6c54008) at app_macro.c:486
ASTERISK-13 0x080bdbe8 in pbx_exec (c=0x82557f8, app=0x826cba0, data=0xb6c54008) at pbx.c:537
ASTERISK-14 0x080c1000 in pbx_extension_helper (c=0x82557f8, con=0x0, context=0x8255978 "macro-vis-incoming", exten=0x82559c8 "s",
   priority=1, label=0x0, callerid=0x82fead8 "021585404", action=E_SPAWN) at pbx.c:1863
ASTERISK-15 0x080c20b2 in ast_spawn_extension (c=0x82557f8, context=0x8255978 "macro-vis-incoming", exten=0x82559c8 "s", priority=1,
   callerid=0x82fead8 "021585404") at pbx.c:2318
ASTERISK-16 0x080c24b8 in __ast_pbx_run (c=0x82557f8) at pbx.c:2407
ASTERISK-17 0x080c328a in pbx_thread (data=0x82557f8) at pbx.c:2622
ASTERISK-18 0x08100ac9 in dummy_start (data=0x81e3ce8) at utils.c:850
ASTERISK-19 0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-20 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 3 (process 27949):
#0  0xb7f3c410 in ?? ()
#1  0xb6d13788 in ?? ()
#2  0xffffffff in ?? ()
---Type <return> to continue, or q <return> to quit---
#3  0x00000004 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080827e9 in ast_waitfor_nandfds (c=0xb6d13a04, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0xb6d13a00)
   at channel.c:1760
ASTERISK-2  0x08082b19 in ast_waitfor_n (c=0xb6d13a04, n=2, ms=0xb6d13a00) at channel.c:1822
ASTERISK-3  0x080893be in ast_generic_bridge (c0=0x830fd08, c1=0x82e8608, config=0xb6d1419c, fo=0xb6d13f30, rc=0xb6d13f2c, bridge_end=
     {tv_sec = 0, tv_usec = 0}) at channel.c:3948
ASTERISK-4  0x0808a8c0 in ast_channel_bridge (c0=0x830fd08, c1=0x82e8608, config=0xb6d1419c, fo=0xb6d13f30, rc=0xb6d13f2c)
   at channel.c:4286
ASTERISK-5  0xb7ca1bbb in ast_bridge_call (chan=0x830fd08, peer=0x82e8608, config=0xb6d1419c) at res_features.c:1570
ASTERISK-6 0xb7bd8376 in dial_exec_full (chan=0x830fd08, data=0xb6d16f28, peerflags=0xb6d14df4, continue_exec=0x0) at app_dial.c:1835
ASTERISK-7 0xb7bd85c2 in dial_exec (chan=0x830fd08, data=0xb6d16f28) at app_dial.c:1882
ASTERISK-8 0x080bdbe8 in pbx_exec (c=0x830fd08, app=0x819ae70, data=0xb6d16f28) at pbx.c:537
ASTERISK-9 0x080c1000 in pbx_extension_helper (c=0x830fd08, con=0x0, context=0x830fe88 "macro-vis-incoming", exten=0x830fed8 "s",
   priority=4, label=0x0, callerid=0x82c9f80 "073789093", action=E_SPAWN) at pbx.c:1863
ASTERISK-10 0x080c20b2 in ast_spawn_extension (c=0x830fd08, context=0x830fe88 "macro-vis-incoming", exten=0x830fed8 "s", priority=4,
   callerid=0x82c9f80 "073789093") at pbx.c:2318
ASTERISK-11 0xb6f091d6 in _macro_exec (chan=0x830fd08, data=0xb6d1c008, exclusive=0) at app_macro.c:308
ASTERISK-12 0xb6f09eef in macro_exec (chan=0x830fd08, data=0xb6d1c008) at app_macro.c:486
ASTERISK-13 0x080bdbe8 in pbx_exec (c=0x830fd08, app=0x826cba0, data=0xb6d1c008) at pbx.c:537
ASTERISK-14 0x080c1000 in pbx_extension_helper (c=0x830fd08, con=0x0, context=0x830fe88 "macro-vis-incoming", exten=0x830fed8 "s",
   priority=1, label=0x0, callerid=0x82c9f80 "073789093", action=E_SPAWN) at pbx.c:1863
ASTERISK-15 0x080c20b2 in ast_spawn_extension (c=0x830fd08, context=0x830fe88 "macro-vis-incoming", exten=0x830fed8 "s", priority=1,
   callerid=0x82c9f80 "073789093") at pbx.c:2318
ASTERISK-16 0x080c24b8 in __ast_pbx_run (c=0x830fd08) at pbx.c:2407
ASTERISK-17 0x080c328a in pbx_thread (data=0x830fd08) at pbx.c:2622
ASTERISK-18 0x08100ac9 in dummy_start (data=0x82c9ae8) at utils.c:850
ASTERISK-19 0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-20 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 2 (process 27964):
#0  0xb7f3c410 in ?? ()
#1  0xb6d8f3c8 in ?? ()
#2  0xffffffff in ?? ()
#3  0x00000004 in ?? ()
#4  0xb7e318f3 in poll () from /lib/tls/i686/cmov/libc.so.6
ASTERISK-1  0x080827e9 in ast_waitfor_nandfds (c=0xb6d8f644, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0xb6d8f640)
   at channel.c:1760
ASTERISK-2  0x08082b19 in ast_waitfor_n (c=0xb6d8f644, n=2, ms=0xb6d8f640) at channel.c:1822
ASTERISK-3  0x080893be in ast_generic_bridge (c0=0x82cae80, c1=0x829ebd8, config=0xb6d9151c, fo=0xb6d8fb70, rc=0xb6d8fb6c, bridge_end=
     {tv_sec = 0, tv_usec = 0}) at channel.c:3948
ASTERISK-4  0x0808a8c0 in ast_channel_bridge (c0=0x82cae80, c1=0x829ebd8, config=0xb6d9151c, fo=0xb6d8fb70, rc=0xb6d8fb6c)
   at channel.c:4286
ASTERISK-5  0xb7ca1bbb in ast_bridge_call (chan=0x82cae80, peer=0x829ebd8, config=0xb6d9151c) at res_features.c:1570
ASTERISK-6 0xb74eaed5 in try_calling (qe=0xb6d91850, options=0xb6d917fc "", announceoverride=0x0, url=0x0, tries=0xb6d919f0,
   noption=0xb6d919ec, agi=0x0) at app_queue.c:3154
---Type <return> to continue, or q <return> to quit---
ASTERISK-7 0xb74ee0b9 in queue_exec (chan=0x82cae80, data=0xb6d91b1c) at app_queue.c:3999
ASTERISK-8 0x080bdbe8 in pbx_exec (c=0x82cae80, app=0x81e8c08, data=0xb6d91b1c) at pbx.c:537
ASTERISK-9 0xb6e5474f in realtime_exec (chan=0x82cae80, context=0x82cb000 "routing", exten=0x82cb050 "queue-IRD_Help", priority=3,
   callerid=0x82cb3e8 "0", data=0x82a55ad "routing@extensions") at pbx_realtime.c:216
ASTERISK-10 0x080c10dc in pbx_extension_helper (c=0x82cae80, con=0x0, context=0x82cb000 "routing", exten=0x82cb050 "queue-IRD_Help",
   priority=3, label=0x0, callerid=0x82cb3e8 "0", action=E_SPAWN) at pbx.c:1874
ASTERISK-11 0x080c20b2 in ast_spawn_extension (c=0x82cae80, context=0x82cb000 "routing", exten=0x82cb050 "queue-IRD_Help", priority=3,
   callerid=0x82cb3e8 "0") at pbx.c:2318
ASTERISK-12 0x080c24b8 in __ast_pbx_run (c=0x82cae80) at pbx.c:2407
ASTERISK-13 0x080c328a in pbx_thread (data=0x82cae80) at pbx.c:2622
ASTERISK-14 0x08100ac9 in dummy_start (data=0x8276870) at utils.c:850
ASTERISK-15 0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-16 0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 1 (process 27734):
#0  0x0808157c in ast_channel_datastore_free (datastore=0x82ca388) at channel.c:1341
#1  0x08081122 in ast_channel_free (chan=0x82e6378) at channel.c:1243
#2  0x08081ec6 in ast_hangup (chan=0x82e6378) at channel.c:1553
#3  0x080c30be in __ast_pbx_run (c=0x82e6378) at pbx.c:2562
#4  0x080c328a in pbx_thread (data=0x82e6378) at pbx.c:2622
ASTERISK-1  0x08100ac9 in dummy_start (data=0x8296660) at utils.c:850
ASTERISK-2  0xb7f20240 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
ASTERISK-3  0xb7e3b49e in clone () from /lib/tls/i686/cmov/libc.so.6
By: Matt Riddell (zx81) 2009-01-13 11:01:15.000-0600

Core dates and times:

2008-12-10T09:02:57+1300
2008-12-16T14:28:07+1300
2008-12-19T16:07:05+1300
2009-01-05T09:11:58+1300
2009-01-06T11:36:32+1300
2009-01-07T11:48:44+1300
2009-01-09T11:56:42+1300
2009-01-12T11:00:31+1300
2009-01-13T11:10:57+1300
2009-01-13T14:30:51+1300
2009-01-13T14:58:27+1300
2009-01-13T16:11:41+1300
2009-01-13T16:20:26+1300

You can see yesterday was the worst (13th)
By: Mark Michelson (mmichelson) 2009-01-13 11:04:53.000-0600

The new crash seems to be another case of the same thing happening, just in a new place now. When a channel is being hung up, we're trying to free a datastore that already was previously freed. The problem is that when we freed it previously, we needed to also remove the datastore from the list so that this wouldn't happen again.

The new backtrace doesn't give a lot of information regarding which datastore carries the problem, although assuming the queue transfer datastore is at fault is not a bad base assumption.

ZX81, I have two requests. One is that for future backtrace uploads, please attach the text as a file upload instead of placing the text directly in notes. The second is, if possible, could you run Asterisk under valgrind? Instructions for how to do it are in doc/valgrind.txt in the Asterisk source directory.

I'll continue looking myself and trying my own valgrind runs, too.
By: Matt Riddell (zx81) 2009-01-13 11:46:50.000-0600

K, will do and am running in valgrind
By: Mark Michelson (mmichelson) 2009-01-13 11:57:54.000-0600

I have uploaded 14086v2.patch which adds some extra locking around datastore operations in app_queue. This patch also contains the contents of 14086.patch, so there is no need to apply them both. Please try the new patch and let me know if the crashes still occur.
By: Matt Riddell (zx81) 2009-01-13 12:21:31.000-0600

Patch in place with traffic starting - will update
By: Matt Riddell (zx81) 2009-01-13 12:28:18.000-0600

Agents starting to log in
By: Matt Riddell (zx81) 2009-01-14 01:43:44.000-0600

Ok, got through the day with no crashes at all.

While it's possible that the problem still exists, we had 5 crashes yesterday, so I'd say it's pretty likely it has gone.

You've got my support on closing this one out, maybe festr wants to try it out too.

If I see the same problem again I can catch you on IRC and get you to reopen it.

Thanks heaps for your attention to this putnopvut!
By: Martin Vit (festr) 2009-01-14 03:03:41.000-0600

Please let this open until confirmation that problem is gone on my system. I'm going to deploy patch and test. I have 20 crashes per day.
By: Martin Vit (festr) 2009-01-14 10:33:20.000-0600

Ok, my production system survived 480 calls without crash today.
By: Kevin Scott Adams (nivek) 2009-01-14 10:53:11.000-0600

I have added the patch to my dev system and have been able to crash every time on an attended transfer.

bACKtRACE'S reveal:
#0  0x00627410 in __kernel_vsyscall ()
#1  0x45713ee9 in raise () from /lib/libc.so.6
#2  0x457154f1 in abort () from /lib/libc.so.6
#3  0x4574853b in __libc_message () from /lib/libc.so.6
#4  0x4574fa68 in _int_free () from /lib/libc.so.6
ASTERISK-1  0x45752f6f in free () from /lib/libc.so.6
ASTERISK-2  0x00b02aaa in queue_transfer_destroy (data=0xb7d00058) at app_queue.c:2813
ASTERISK-3  0x08087246 in ast_channel_datastore_free (datastore=0xb7d53a00) at channel.c:1323
ASTERISK-4  0x080869cb in ast_channel_free (chan=0x8f91ff8) at channel.c:1224
ASTERISK-5  0x08087c4a in ast_hangup (chan=0x8f91ff8) at channel.c:1536
ASTERISK-6 0x080d621f in __ast_pbx_run (c=0x8f91ff8) at pbx.c:2561
ASTERISK-7 0x080d646f in pbx_thread (data=0x8f91ff8) at pbx.c:2621
ASTERISK-8 0x08119d76 in dummy_start (data=0x8f68b08) at utils.c:912
ASTERISK-9 0x458c6433 in start_thread () from /lib/libpthread.so.0
ASTERISK-10 0x457b4a1e in clone () from /lib/libc.so.6

AND

#0  0x08087221 in ast_channel_datastore_free (datastore=0xb7d660a8) at channel.c:1322
1322            if (datastore->info->destroy != NULL && datastore->data != NULL) {
(gdb) bt
#0  0x08087221 in ast_channel_datastore_free (datastore=0xb7d660a8) at channel.c:1322
#1  0x080869cb in ast_channel_free (chan=0x9947590) at channel.c:1224
#2  0x08087c4a in ast_hangup (chan=0x9947590) at channel.c:1536
#3  0x080d621f in __ast_pbx_run (c=0x9947590) at pbx.c:2561
#4  0x080d646f in pbx_thread (data=0x9947590) at pbx.c:2621
ASTERISK-1  0x08119d76 in dummy_start (data=0x9942590) at utils.c:912
ASTERISK-2  0x458c6433 in start_thread () from /lib/libpthread.so.0
ASTERISK-3  0x457b4a1e in clone () from /lib/libc.so.6

AND

#0  0x08087221 in ast_channel_datastore_free (datastore=0xb7d065b8) at channel.c:1322
1322            if (datastore->info->destroy != NULL && datastore->data != NULL) {
(gdb) bt
#0  0x08087221 in ast_channel_datastore_free (datastore=0xb7d065b8) at channel.c:1322
#1  0x080869cb in ast_channel_free (chan=0xb7d27460) at channel.c:1224
#2  0x08087c4a in ast_hangup (chan=0xb7d27460) at channel.c:1536
#3  0x080d621f in __ast_pbx_run (c=0xb7d27460) at pbx.c:2561
#4  0x080d646f in pbx_thread (data=0xb7d27460) at pbx.c:2621
ASTERISK-1  0x08119d76 in dummy_start (data=0xb7d0c1f8) at utils.c:912
ASTERISK-2  0x458c6433 in start_thread () from /lib/libpthread.so.0
ASTERISK-3  0x457b4a1e in clone () from /lib/libc.so.6

Reverting back to new_chan instead of old_chan has resulted in no crashes.
By: Kevin Scott Adams (nivek) 2009-01-14 11:01:33.000-0600

Sorry, I should mention that it is on hangup of the transferred caller.
By: Martin Vit (festr) 2009-01-14 11:15:38.000-0600

nivek: what patch did you applied? 14086.patch or 14086v2.patch?
By: Kevin Scott Adams (nivek) 2009-01-14 11:16:23.000-0600

Sorry, one more thing.  We are doing all attended transfers.  No blind transfers!!!
By: Kevin Scott Adams (nivek) 2009-01-14 11:20:27.000-0600

Crashes occurred with both patches.
By: Mark Michelson (mmichelson) 2009-01-14 12:04:18.000-0600

Let's clear up a few things here:

First off, to ZX81 and festr, I know when we talked on IRC yesterday, both of you discussed running valgrind. Were you running valgrind when you were not having your crashes? It may be that valgrind was preventing the crashes from occurring but they would have occurred when not running it. I would like to see the output from those valgrind runs if you were running it.

nivek: if ZX81 and festr do not have valgrind output from their test runs from yesterday, could you please run Asterisk under valgrind? Instructions for how to do so are in doc/valgrind.txt in the Asterisk source directory.

Thanks.
By: Kevin Scott Adams (nivek) 2009-01-14 13:01:39.000-0600

putnopvut,

What from the valgrind would you like??
By: Kevin Scott Adams (nivek) 2009-01-14 13:05:21.000-0600

1231958718 - New session
1231959226 - New session
WARNING: Freeing unused memory at 0x4de0f00, in ast_channel_datastore_free of channel.c, line 1334
WARNING: Freeing unused memory at 0x41e1ca8, in ast_channel_datastore_free of channel.c, line 1334
1231959483 - New session


This is from /var/log/asterisk/mmlog
By: Kevin Scott Adams (nivek) 2009-01-14 13:07:28.000-0600

Here is from valgrind.txt that last line basically repeats itself.


==22682==    by 0x458C6432: start_thread (in /lib/libpthread-2.4.so)
==22682==    by 0x457B4A1D: clone (in /lib/libc-2.4.so)
==22682==  Address 0x4DE0F04 is 108 bytes inside a block of size 128 free'd
==22682==    at 0x4004E41: free (vg_replace_malloc.c:235)
==22682==    by 0x807447C: __ast_free_region (astmm.c:187)
==22682==    by 0x8074E6E: __ast_free (astmm.c:221)
==22682==    by 0x8089DF0: ast_channel_datastore_free (channel.c:1334)
==22682==    by 0x46165BD: ??? (app_queue.c:3493)
==22682==    by 0x4619BB7: ??? (app_queue.c:4325)
==22682==    by 0x80D414C: pbx_exec (pbx.c:537)
==22682==    by 0x80D7ED1: pbx_extension_helper (pbx.c:1862)
==22682==    by 0x80D925E: ast_spawn_extension (pbx.c:2317)
==22682==    by 0x80D96A6: __ast_pbx_run (pbx.c:2406)
==22682==    by 0x80DA4B9: pbx_thread (pbx.c:2621)
==22682==    by 0x811E29F: dummy_start (utils.c:912)
==22682==
==22682== Invalid read of size 4
==22682==    at 0x8089D9A: ast_channel_datastore_free (channel.c:1328)
==22682==    by 0x80894C9: ast_channel_free (channel.c:1224)
==22682==    by 0x808A7C0: ast_hangup (channel.c:1536)
==22682==    by 0x80DA251: __ast_pbx_run (pbx.c:2561)
==22682==    by 0x80DA4B9: pbx_thread (pbx.c:2621)
==22682==    by 0x811E29F: dummy_start (utils.c:912)
==22682==    by 0x458C6432: start_thread (in /lib/libpthread-2.4.so)
==22682==    by 0x457B4A1D: clone (in /lib/libc-2.4.so)
==22682==  Address 0x4DE0F00 is 104 bytes inside a block of size 128 free'd
==22682==    at 0x4004E41: free (vg_replace_malloc.c:235)
==22682==    by 0x807447C: __ast_free_region (astmm.c:187)
==22682==    by 0x8074E6E: __ast_free (astmm.c:221)
==22682==    by 0x8089DF0: ast_channel_datastore_free (channel.c:1334)
==22682==    by 0x46165BD: ??? (app_queue.c:3493)
==22682==    by 0x4619BB7: ??? (app_queue.c:4325)
==22682==    by 0x80D414C: pbx_exec (pbx.c:537)
==22682==    by 0x80D7ED1: pbx_extension_helper (pbx.c:1862)
==22682==    by 0x80D925E: ast_spawn_extension (pbx.c:2317)
==22682==    by 0x80D96A6: __ast_pbx_run (pbx.c:2406)
==22682==    by 0x80DA4B9: pbx_thread (pbx.c:2621)
==22682==    by 0x811E29F: dummy_start (utils.c:912)
==22793== Warning: invalid file descriptor 1014 in syscall close()
==22793== Warning: invalid file descriptor 1015 in syscall close()
==22793== Warning: invalid file descriptor 1016 in syscall close()
==22793== Warning: invalid file descriptor 1017 in syscall close()
By: Kevin Scott Adams (nivek) 2009-01-14 13:11:36.000-0600

Restoring this patch
       if ((datastore = ast_channel_datastore_find(old_chan, &queue_transfer_info, NULL))) {
               ast_channel_datastore_remove(old_chan, datastore);


to

       if ((datastore = ast_channel_datastore_find(new_chan, &queue_transfer_info, NULL))) {
               ast_channel_datastore_remove(new_chan, datastore);


and I have not crashes!!!

One thing...I took me 3 try's to get it to crash.  The first two under a 'stop now' command the last was a 'restart now'.

Hope this helps.

-nevik
By: Mark Michelson (mmichelson) 2009-01-14 13:30:04.000-0600

nivek: Here is a very important question for you: what version of Asterisk are you using? I realize that the patch here would actually *cause* problems if you are using a released version instead of what is in the current 1.4 svn branch. The reason for this is due to a behavior change that was introduced with channel masquerades that was not properly updated in app_queue. The patch I attached here accounts for that change in channel masquerade behavior, but would actually cause problems if applied to a version of Asterisk which did not have the masquerade behavior change (which includes releases up to 1.4.22).
By: Kevin Scott Adams (nivek) 2009-01-14 14:25:02.000-0600

Yes.  I am at 1.4.22!  Sorry!
By: Mark Michelson (mmichelson) 2009-01-14 14:27:18.000-0600

Phew, that's a relief! :)

All right, so based on both ZX81 and festr reporting that there are no crashes, I'll wait until the end of my working day (GMT 23:30, 14 Jan) and commit the patch if I hear nothing new before then.
By: Matt Riddell (zx81) 2009-01-14 14:28:06.000-0600

Nah, in the end I put my faith in your patch and ran without valgrind - worried also about degradation. So the lack of crashes was without valgrind.  I also upgraded to latest 1.4 svn as the second patch had the last two chunks fail without doing so.
By: Kevin Scott Adams (nivek) 2009-01-14 14:29:04.000-0600

Sorry about that I did not know there was underlying changes to channel masquerades.  Only reason I interjected was because of my change.  I thought I did something wrong with my original patch (No Karma for me)
By: Mark Michelson (mmichelson) 2009-01-14 14:30:19.000-0600

nivek: there really was no way you could have known about the underlying changes unless you have been monitoring Asterisk commits closely, so don't feel bad about it.
By: Mark Michelson (mmichelson) 2009-01-14 18:09:09.000-0600

Just letting everyone know that I'll be committing the patch here to 1.4 now. Thank you all very much for providing me with the information I needed to fix this.
By: Digium Subversion (svnbot) 2009-01-14 18:11:02.000-0600

Repository: asterisk
Revision: 168628

U   branches/1.4/apps/app_queue.c

------------------------------------------------------------------------
r168628 | mmichelson | 2009-01-14 18:11:02 -0600 (Wed, 14 Jan 2009) | 16 lines

Fix some crashes from bad datastore handling in app_queue.c

* The queue_transfer_fixup function was searching for and removing
 the datastore from the incorrect channel, so this was fixed.

* Most datastore operations regarding the queue_transfer datastore
 were being done without the channel locked, so proper channel locking
 was added, too.

(closes issue ASTERISK-13226)
Reported by: ZX81
Patches:
     14086v2.patch uploaded by putnopvut (license 60)
Tested by: ZX81, festr


------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=168628
By: Digium Subversion (svnbot) 2009-01-14 18:14:18.000-0600

Repository: asterisk
Revision: 168629

_U  trunk/
U   trunk/apps/app_queue.c

------------------------------------------------------------------------
r168629 | mmichelson | 2009-01-14 18:14:17 -0600 (Wed, 14 Jan 2009) | 24 lines

Merged revisions 168628 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r168628 | mmichelson | 2009-01-14 18:11:01 -0600 (Wed, 14 Jan 2009) | 16 lines

Fix some crashes from bad datastore handling in app_queue.c

* The queue_transfer_fixup function was searching for and removing
 the datastore from the incorrect channel, so this was fixed.

* Most datastore operations regarding the queue_transfer datastore
 were being done without the channel locked, so proper channel locking
 was added, too.

(closes issue ASTERISK-13226)
Reported by: ZX81
Patches:
     14086v2.patch uploaded by putnopvut (license 60)
Tested by: ZX81, festr


........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=168629
By: Digium Subversion (svnbot) 2009-01-14 18:14:49.000-0600

Repository: asterisk
Revision: 168630

_U  branches/1.6.0/
U   branches/1.6.0/apps/app_queue.c

------------------------------------------------------------------------
r168630 | mmichelson | 2009-01-14 18:14:49 -0600 (Wed, 14 Jan 2009) | 32 lines

Merged revisions 168629 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r168629 | mmichelson | 2009-01-14 18:14:17 -0600 (Wed, 14 Jan 2009) | 24 lines

Merged revisions 168628 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r168628 | mmichelson | 2009-01-14 18:11:01 -0600 (Wed, 14 Jan 2009) | 16 lines

Fix some crashes from bad datastore handling in app_queue.c

* The queue_transfer_fixup function was searching for and removing
 the datastore from the incorrect channel, so this was fixed.

* Most datastore operations regarding the queue_transfer datastore
 were being done without the channel locked, so proper channel locking
 was added, too.

(closes issue ASTERISK-13226)
Reported by: ZX81
Patches:
     14086v2.patch uploaded by putnopvut (license 60)
Tested by: ZX81, festr


........

................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=168630
By: Digium Subversion (svnbot) 2009-01-14 18:15:23.000-0600

Repository: asterisk
Revision: 168631

_U  branches/1.6.1/
U   branches/1.6.1/apps/app_queue.c

------------------------------------------------------------------------
r168631 | mmichelson | 2009-01-14 18:15:23 -0600 (Wed, 14 Jan 2009) | 32 lines

Merged revisions 168629 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r168629 | mmichelson | 2009-01-14 18:14:17 -0600 (Wed, 14 Jan 2009) | 24 lines

Merged revisions 168628 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r168628 | mmichelson | 2009-01-14 18:11:01 -0600 (Wed, 14 Jan 2009) | 16 lines

Fix some crashes from bad datastore handling in app_queue.c

* The queue_transfer_fixup function was searching for and removing
 the datastore from the incorrect channel, so this was fixed.

* Most datastore operations regarding the queue_transfer datastore
 were being done without the channel locked, so proper channel locking
 was added, too.

(closes issue ASTERISK-13226)
Reported by: ZX81
Patches:
     14086v2.patch uploaded by putnopvut (license 60)
Tested by: ZX81, festr


........

................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=168631