[Home]

Summary:ASTERISK-14304: [patch] Seg fault in chan_local - local_pvt_destroy
Reporter:Sebastian (sroberts)Labels:
Date Opened:2009-06-11 04:50:21Date Closed:2010-05-25 10:50:47
Priority:CriticalRegression?No
Status:Closed/CompleteComponents:Channels/chan_local
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) backtraces_for_note_0109008.txt
( 1) gdb-backtrace.txt
( 2) gdb-backtrace2.txt
( 3) Issue15314_Move_Nulling_owner.patch
( 4) sticking-plaster-1.txt
Description:This is the same issue as 14780. We also use SNOM phones (300s and 320s) however these extensions are not connected to the server on which Asterisk crashed. The crash occurred on the queue server.

A backtrace of the crash has been attached.

The crash here occurred when callfile finished execution. We use callfiles to pause/unpause the agents. The local_pvt being freed is not null:

(gdb) frame 2
#2  0x002dd837 in local_pvt_destroy (pvt=0xa23e928) at chan_local.c:159
159             free(pvt);
(gdb) p pvt
$1 = (struct local_pvt *) 0xa23e928
(gdb) p *pvt
$2 = {lock = {mutex = {__m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 1, __m_lock = {__status = 0, __spinlock = 0}}, track = 1, file = {0x2e0f08 "chan_local.c", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, lineno = {158, 0, 0, 0, 0, 0, 0, 0, 0, 0}, reentrancy = 0, func = {0x2e0fbe "local_pvt_destroy", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, thread = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, reentr_mutex = {__m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 1, __m_lock = {__status = 0, __spinlock = 0}}}, flags = 16, context = "vital-out", '\0' <repeats 70 times>, exten = "*\000vital-out\000n", '\0' <repeats 66 times>, reqformat = 64, owner = 0x0, chan = 0x0, u_owner = 0xa2234c8, u_chan = 0xa1c3500, list = {next = 0x0}}


Due to the fact that our queue servers are so busy I cannot simply upgrade it to a newer version (this one in particular handles around 10000 calls per day) unless I know a version is stable. I've tested 1.4.25 and it proved horrendously unstable (deadlocks and seg faults).



****** ADDITIONAL INFORMATION ******

OS: CentOS 4.7
Server: HP DL360 with 2x Intel Xeon E5420 2.5Ghz processors, 2 Gb RAM
Comments:By: Sebastian (sroberts) 2009-06-11 05:09:25

I've attached gdb-backtrace2.txt as well. This is the backtrace from a different server who's Asterisk crashed yesterday. This time the crash was in ast_cdr_free, while trying to free the cdr struct. I'm posting this here as well because it was trying to free a cdr which was created by a callfile which leads me to wonder whether there isn't perhaps an issue with the callfile handling? This server also handles around 10000 calls per day and both can have up to 150 agents logged in at peak times.

One more thing to note is that the call files execute dialplan logic which was written in AEL, while all other dialplan scripting is written in normal Asterisk dialplan style. I don't know whether this would make a difference, but I'm just mentioning it nonetheless.
By: David Woolley (davidw) 2009-08-13 11:30:09

We have a variation on this in 1.6.1.0.  The actual stack trace goes through dial_exec_full, rather than the route this one takes.  It is reproducible, but not reliably so (we've not been able to reproduce it on the development systems, but it was reasonably re-producable on the load test rig).

We have tried the ASTERISK-13856 patch.  This seems to reduce the window, but not remove it, making it even more difficult to reproduce - it originally only showed on a real machine, whereas most testing is done on VMWare guests.  In this case, the private structure has already been invalidated before the first lock attempt in hangup_local, so that call produces an invalid argument error from pthread_mutex_lock (we have lock debugging compiled in).  I breakpointed the call to log the locking error and confirmed that owner was already 0 in the private structure.

Note that this means that this can no longer be considered duplicate, as the final crash backtrace is indistinguishable whether or not the lock failure has already occured.

At the moment our priorities are to put in defensive coding to reduce but not remove the window, but I'll try and come back and provide the traces and see if I can audit local channel's locking.
By: Sebastian (sroberts) 2009-08-14 06:58:24

Thanks, giving it a bash now.
By: David Woolley (davidw) 2009-08-14 07:00:29

We tried the defensive code in sticking-plaster-1, but whilst it avoided the locking failures and the double free, it has just moved the problem.  Now it is presenting as local_devicestate accessing a broken (0x700) pointer to the private structure, in the device state thread, so I presume that a local channel has been freed whilst there is still a live reference.

Note the patch is not recommended for use, as it only attacks a symptom, not the real problem.  Also it may apply with an offset, as there are non-locking related patches in the code we are trying to release.
By: David Woolley (davidw) 2009-08-14 07:02:32

sroberts:  I added that patch more as documentation of what we'd tried than as something that should actually be used by others.  We also have the ASTERISK-13856 patch in.
By: Sebastian (sroberts) 2009-08-14 07:11:13

yeah, I see. I'm getting deadlocks in local_hangup at the moment as well so I'm busy testing a bunch of different ideas for avoiding the seg-fault and deadlocks within local_hangup.

In any event, it is possible that p or p->owner can be destroyed later on in the function anyway.


By: David Woolley (davidw) 2009-08-14 07:16:12

We haven't had deadlocks (on 1.6.1.0).  If you are using a similar version, do you know where it is deadlocking?

(I suspect the problems here all arise from the fact that the private structure can be approached form either parent, but there is no clear correct locking sequence.)
By: David Woolley (davidw) 2009-08-14 10:34:48

Having looked at the locking strategy, I think the sticking-plaster patch is wrong; it should not be doing null tests on owner and chan, as they can be independently null.

Unfortunately, some commercial constraints mean that access to the machine on which we can reliably break this, and the personnel that can do so, may have become limited, so I may not be able to retry quickly.
By: David Woolley (davidw) 2009-08-14 10:38:54

I'm told that setting the kernel parameter maxcpus to 1 significantly reduces the crash frequency.  I suggested trying something like this on the basis that the VMWare model was difficult to crash, and I was speculating that Asterisk rarely got more than one CPU core.  We are still talking about an unacceptable rate, though.
By: David Woolley (davidw) 2009-08-14 11:29:23

In our case, using 1.6.1.0, with the ASTERISK-13856 patch in, the problem has to arise in the following lines:

599 :     p->chan = NULL;
600 :     ast_clear_flag(p, LOCAL_LAUNCHED_PBX);
601 :     ast_module_user_remove(p->u_chan);
602 :     } else {
603 :     p->owner = NULL;
604 :     ast_module_user_remove(p->u_owner);
605 :     while (p->chan && ast_channel_trylock(p->chan)) {
606 :     DEADLOCK_AVOIDANCE(&p->lock);
607 :     }
608 :     if (p->chan) {
609 :     ast_queue_hangup(p->chan);
610 :     ast_channel_unlock(p->chan);
611 :     }
612 :     }
613 :    
614 :     ast->tech_pvt = NULL;

This is because both owner and chan are null, but the owner side tech_pvt isn't.  It pretty much has to be the DEADLOCK_AVOIDANCE.  It is temporarily releasing the lock on the private structure.

Is there any good reason, except possibly a misguided attempt to factor out common code, for the clearing of tech_pvt not to be done immediately after the clearing of the owner/chan?

Although this is the only place where that combination can occur, I still cannot explain how local hangup can get re-entered on the owner channel without a recursive call on the stack....
By: David Woolley (davidw) 2009-08-14 13:43:11

On second thoughts, as local_hangup doesn't yet have the private structure lock, it could have read the private structure pointer before it got wiped, and the real problem is how the owner pointer got cleared if the owner channel was locked.

I'd like to check whether ast->tech_pvt was still valid when the bad lock occured, but that means getting gdb, the libraries and the dump all on the same machine.
By: David Woolley (davidw) 2009-08-17 07:16:55

Unless there are constraints that really should be commented, the following looks unsafe.  In all cases the "if (ast)" test will always succeed.

   582                         /* Deadlock avoidance */
   583                         while (p->owner && ast_channel_trylock(p->owner)) {
   584                                 ast_mutex_unlock(&p->lock);
   585                                 if (ast) {
   586                                         ast_channel_unlock(ast);
   587                                 }
   588                                 usleep(1);
   589                                 if (ast) {
   590                                         ast_channel_lock(ast);
   591                                 }
   592                                 ast_mutex_lock(&p->lock);
   593                         }

On the usleep, all the locks are off, so if this can be re-entered for the chan side, p and ast may cease to point to valid memory.

I can't (yet) work out a scenario where this would cause the current problem, though.
By: David Woolley (davidw) 2009-08-17 09:50:55

At the breakpoint on the mutex_lock error path (see note 109008), the local channel's (owner side) tech_pvt pointer hasn't been cleared:

(gdb) print *ast
$1 = {tech = 0x2983e0, tech_pvt = 0x889faa0, music_state = 0x0,

even though both pointers have been cleared in the other direction (note that the full print *p output is embedded in the back trace):

(gdb)   print *p
$4 = {lock = {track = {file = {0x829408 "", 0x829408 "", 0x0, 0x0, 0x0, 0x0,
.....
   impl = '\0' <repeats 11 times>}, owner = 0x0, chan = 0x0,

The lock debugging traces seem to indicate that it has already gone through local_pvt_destroy:

     reentrancy = 0, func = {0x296b85 "local_pvt_destroy", 0x0, 0x0, 0x0,
       0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, thread = {0, 0, 0, 0, 0, 0, 0, 0, 0,
       0}, backtrace = {{addresses = {0x80fb75c, 0x292808, 0x2956e6,

which is not altogether surprising, given the way it has failed.
By: Sebastian (sroberts) 2009-08-17 09:57:54

I may not be following you here, but is it perhaps because the Local channel is just a proxy for an underlying channel. I know there is a fix out for 1.4.26 which is supposed to prevent the PVT structure from being cleared out from underneath the local channel. This fix is only for IAX channels though. It would make sense though, since in local_hangup the code surrenders the lock it has on ast AND p->lock, giving ample time for the IAX channel underneath to grab those resources and free them. The patch is for issue 15377 and is included in 1.4.26.

I'm still running 1.4.22, however I have applied this fix and it seems to be holding for now. I don't know whether this has been implemented in 1.6 though
By: David Woolley (davidw) 2009-08-17 10:06:18

I'll have a look, but the local channel is rather more than a proxy in the way it is implemented and p->lock should only be accessed by chan_local.c, so the local channel should be taking responsbility for its safe use.

Agent is closer to being a true proxy.

(Current line of thought - is IS_OUTBOUND safe?)


By: David Woolley (davidw) 2009-08-17 10:53:17

I cannot find local_hangup on the stack in any other thread, which is making me wonder whether p->owner has actually been cleared by the chan side hangup code, being re-entered and getting confused because IS_OUTBOUND is only valid whilst the chan side is still open.

If two threads could reach the initial channel lock for the chan side channel, both trying to do a hangup, it would be possible for IS_OUTBOUND to get confused and have the second one through run the owner side code.  As it would still have the ast value for the chan side, this would lead to the owner's tech_pvt still being set.
By: David Woolley (davidw) 2009-08-17 11:10:31

In this case, the owner side doesn't have a CHANLOCALSTATUS variable set, which makes it unlikely that the unsafe lock releases mentioned above are a factor here, although this is possibly not completely certain given that p->owner seems to have been inappropriately cleared.

Possibly more significant, there is no DIALSTATUS carried over that might have been inherited.  The chan side was going to do a Queue, rather than Dial, but in this case probably didn't do this, because the dialplan logic probably decided that out policy for accepting for queuing were violated.

The ;2 side, itself, is almost certainly closed by now.
By: David Woolley (davidw) 2009-08-18 06:24:34

I've looked at our logs and this example is actually one where the Queue application was entered.  The last event on the ;2 side of the channel was exitted non-zero from the Queue.  As the calls are test calls, which never volutarily hangup, the Queued call must have been answered.  However it is possible that the local channel was hungup from the ;1 side, as the result of the SIP channel from which it was bridged being AMI parked.

I need to analyze the logs in some more detail, to make sure of the exact sequence.
By: David Woolley (davidw) 2009-08-18 07:36:22

OK. The context is:

Incoming SIP call bridged via Dial to
Problem local channel, which is executing Queue
Queue has been autoanswered by a non-callback Agent
The Agent is owned by another SIP channel.

The CTI has initiated a Park action specifying the incoming SIP call's channel, an effectively infinite timeout, and a channel known to exist as the return channel, which is never intended to be used.

This is the verbose trace leading up to the invalid lock:

[2009-08-13 14:48:52.936] VERBOSE[4109] res_musiconhold.c:     -- Started music on hold, class 'default', on SIP/192.168.71.115-b7406a48

SIP/192.168.71.115-b7406a48 is the incoming SIP call

[2009-08-13 14:48:52.936] VERBOSE[4109] features.c:   == Parked SIP/192.168.71.115-b7406a48 on 701 (lot default). Will timeout back to extension [default] 6050, 50013 in 2000000 seconds

[2009-08-13 14:48:52.936] VERBOSE[4109] pbx.c:     -- Added extension '701' prio
rity 1 to parkedcalls (0x87dd420)

[2009-08-13 14:48:52.937] VERBOSE[4173] res_musiconhold.c:     -- Started music on hold, class 'default', on SIP/sip2-08888cc8

SIP/sip2-08888cc8 is the agent.

[2009-08-13 14:48:52.939] VERBOSE[4173] pbx.c:   == Spawn extension (xxxxxxx, xxxxxx, 8) exited non-zero on 'Local/xxxxxx@xxxxx-b1b8;2'

This is the exit for the Queue application.

The only parameter on the Queue application was "r".

(The local channel is in the chain, because, some time before I got involved with the project, it was discovered that, without it, transferring the caller caused the agent to logout.)
By: David Woolley (davidw) 2009-08-18 10:30:10

Although the incoming SIP channel is a <ZOMBIE>, so the clear is coming from that direction, there is COMPLETECALLER event for the queue in the right second, which indicates that the ;2 to agent bridge was taken down before local_hangup got to the lock attempt.  I'm not sure what is telling it to clear that downstream bridge, but I'm now looking to see if that could generate two hangups.
By: David Woolley (davidw) 2009-08-18 13:26:19

I think I understand why it is being closed from both ends and that it something unusual we do, and probably wrong.  However misuse of AMI shouldn't cause crashes!

We are using Park to simulate hold, and we didn't really want the Channel2 argument of AMI Park at all, as we didn't want any announcements and didn't want the meter maids to do anything.  To satisfy the interface, we gave it the ;2 side of the local channel.

Channel2 gets a soft hangup when the parking completes, which is, presumably, what caused the bridge in Queue to be taken down.

I can't create a crash scenario with a simple collision, so I think there still has to be a double hangup on the ;2 side, which may apply in less unusual cases, and, it really shouldn't crash for an unconventional use of an AMI action!
By: David Woolley (davidw) 2009-08-19 11:15:01

I've also had the same bad mutex at a slightly different place, namely the re-lock of the private structure in the DEADLOCK_AVOIDANCE call.  p->owner, chan null and ast->tech_pvt not null, again.  Same local channel and higher level context.
By: David Woolley (davidw) 2009-08-19 12:33:56

This breakpoint didn't trip, which contra-indicates the theory that IS_OUTBOUND was confused, at least in this case:

2   breakpoint     keep y   0x002f55c0 in local_hangup at chan_local.c:609
       stop only if !p->owner

However, a simple, two party, collision can, with the right timing result in the re-lock in the DEADLOCK_AVOIDANCE failing and subsequent processing on the private structure being invalid.  I don't think this accounts for the failure on the earlier, explicit, lock on p, so I think I still have at least one more thing to discover.  Unfortunately setting breakpoints distorts the timing.

I come back to the question, is there a reason for setting p->owner null before trying for the lock on p->chan?  That may be the easiest way to avoid this variant.
By: David Woolley (davidw) 2009-08-19 13:38:48

Oops.  The breakpoint condition should have been !(p->owner == ast).  However in that particular case, I think it wouldn't have tripped, anyway.

On the other hand, I'm getting double frees without tripping the bad mutex error, so there is maybe a third mode.
By: Sebastian (sroberts) 2009-08-19 13:46:20

I've had similar issues to you, with regards to p->lock being released in the second instance and then being freed during this time. I've rewritten quite a bit of the code in local_hangup to try and avoid this as much as possible. I'm busy testing this at the moment, so if it shows some promising results I'll post the changes I made here tomorrow
By: David Woolley (davidw) 2009-08-20 05:39:25

I think my third instance may be a slight timing variation on the second one.  I think I might try moving the ...owner = NULL, even without confirmation that it is safe, in the hope that I can eliminate one failure mode and concentrate on the one that fails on the explicit lock.

sroberts: if you've done a significant rewrite, what I think would be useful is comments, especially at the beginning, but maybe elsewhere, stating the assertions about locking states and what may be null.
By: David Woolley (davidw) 2009-08-20 06:41:43

We've also been noticing these errors around similar problem areas, and I'm beginning to think they are related:

[2009-08-20 12:12:00.282] ERROR[28492]: pbx.c:8618 device_state_cb: Received invalid event that had no device IE
[2009-08-20 12:12:00.282] ERROR[28492]: app_queue.c:810 device_state_cb: Received invalid event that had no device IE
By: David Woolley (davidw) 2009-08-20 07:09:05

As the device state processing is asynchronous, a bt doesn't directly show the source, howerver it looks like an empty name and a bad pointer:

#0  device_state_cb (event=0xb7200ab8, unused=0x0) at pbx.c:8618
       device = 0xb7200ac4 ""
       sc = (struct statechange *) 0x1
       __PRETTY_FUNCTION__ = "device_state_cb"

(gdb) print *event
$1 = {type = 1280, event_len = 7936, payload = 0xb7200abc ""}
(gdb) print *event->payload
$2 = 0 '\0'

This is 1.6.1.0 with ASTERISK-13856 and with the p->owner = NULL, in local_hangup, moved to immediately after the DEADLOCK_AVOIDANCE, but similar errors have been showing up without the last change.
By: David Woolley (davidw) 2009-08-20 09:29:32

Although I cannot yet eliminate a connection, the device state call with an empty device is being trigged by a later event than the local channel problems, so may be a red herring from that point of view.

It's in an AMI Orinate which is unparking the call whose parking triggered problems with the local channel.
By: David Woolley (davidw) 2009-08-20 10:09:55

The actual device_state_cb errors are the result of a benign bug in ast_pbx_outgoing_cdr_failed or ast_channel_free.  ast_pbx_outgoing_cdr_failed is creating an anonymous channel just to allow it to generate a CDR.  ast_channel_free reports a real device state change on this un-named channel and one eventually gets the errors logged.

Why it is going down this path in the first place is a different question, which may or may not be tortuously related to the local channel problem.

I'm leaving the side issue for now, as I've not got the time to do the due dilligence to ensure that it isn't already known.  If anyone else wants to, please feel welcome.
By: David Woolley (davidw) 2009-08-20 11:37:45

Although the ERROR messages represent a real bug, the reason we went down that path was a red herring.
By: David Woolley (davidw) 2009-08-20 12:10:47

Oops.  I've been confused by the locking failure error message.  It is not reporting the line on which the lock failed, but the line on which it was originally obtained.  It looks like all the failures are on DEADLOCK_AVOIDANCE.

Summary:

1) The DEADLOCK_AVOIDANCE call in the owner branch of local_hangup is unsafe.  Because p->owner has already been nulled, a collision can result in the destruction of the private structure whilst in the DEADLOCK_AVOIDANCE and resultant crash.

2) The deadlock avoidance code in the p->chan branch is potentially unsafe, but no actual failure scenario has been found.

3) The IS_OUTBOUND test is potentially unsafe, but no real life scenario has been found and the only candidate scenarios involve two calls to local_hangup for the chan side, which should not happen.

4) A lock failure in DEADLOCK_AVOIDANCE results in an error message giving a misleading location, probably because it was never envisaged that that lock would fail.  I consider that a bug.

5) A failure to get an answer on the channel side of Originate, and presumably also with call files, will result in ERROR messages from the device state call back, because the routine that generates the CDR uses a dummy channel structure, which is never populated with a name, but channel_free doesn't treat this as a special case.


By: David Woolley (davidw) 2009-08-20 12:53:27

Added patch for item (1) in summary.  Main question is whether there was some reason for nulling p->owner that soon.  Patch is relative to 1.6.1.0.


By: Lott Caskey (lottc) 2009-10-12 15:10:09

I have several highload servers which would crash on a weekly basis...with a near verbatim bt to the attached gdb-backtrace.txt.

I have confirmed that davidw's patch works on 1.6.2.0 on my beta servers and they have been stable for over a month now.  I have also confirmed that the crash returns when the patch is removed.  I now feel confident with moving this version into production within my OSDial project.

The manifestation of the problem was at best unpredictable and highly random, effecting only my high volume systems, something hard to replicate under test conditions.

David, thank-you for your diligence.

Asterisk Team, it would be highly beneficial to schedule his patch for inclusion into the 1.4 / 1.6 branches.
By: Digium Subversion (svnbot) 2009-11-13 13:50:16.000-0600

Repository: asterisk
Revision: 230038

U   branches/1.4/channels/chan_local.c

------------------------------------------------------------------------
r230038 | file | 2009-11-13 13:50:15 -0600 (Fri, 13 Nov 2009) | 9 lines

Fix a crash caused by two threads thinking they should both free the
chan_local private structure when only one should.

(closes issue ASTERISK-14304)
Reported by: sroberts
Patches:
     Issue15314_Move_Nulling_owner.patch uploaded by davidw (license 780)
Tested by: davidw, lottc

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=230038
By: Digium Subversion (svnbot) 2009-11-13 13:51:02.000-0600

Repository: asterisk
Revision: 230039

_U  trunk/
U   trunk/channels/chan_local.c

------------------------------------------------------------------------
r230039 | file | 2009-11-13 13:51:02 -0600 (Fri, 13 Nov 2009) | 16 lines

Merged revisions 230038 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r230038 | file | 2009-11-13 13:44:07 -0600 (Fri, 13 Nov 2009) | 9 lines
 
 Fix a crash caused by two threads thinking they should both free the
 chan_local private structure when only one should.
 
 (closes issue ASTERISK-14304)
 Reported by: sroberts
 Patches:
       Issue15314_Move_Nulling_owner.patch uploaded by davidw (license 780)
 Tested by: davidw, lottc
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=230039
By: Digium Subversion (svnbot) 2009-11-13 13:51:22.000-0600

Repository: asterisk
Revision: 230040

_U  branches/1.6.0/
U   branches/1.6.0/channels/chan_local.c

------------------------------------------------------------------------
r230040 | file | 2009-11-13 13:51:22 -0600 (Fri, 13 Nov 2009) | 23 lines

Merged revisions 230039 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r230039 | file | 2009-11-13 13:44:53 -0600 (Fri, 13 Nov 2009) | 16 lines
 
 Merged revisions 230038 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r230038 | file | 2009-11-13 13:44:07 -0600 (Fri, 13 Nov 2009) | 9 lines
   
   Fix a crash caused by two threads thinking they should both free the
   chan_local private structure when only one should.
   
   (closes issue ASTERISK-14304)
   Reported by: sroberts
   Patches:
         Issue15314_Move_Nulling_owner.patch uploaded by davidw (license 780)
   Tested by: davidw, lottc
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=230040
By: Digium Subversion (svnbot) 2009-11-13 13:51:44.000-0600

Repository: asterisk
Revision: 230041

_U  branches/1.6.1/
U   branches/1.6.1/channels/chan_local.c

------------------------------------------------------------------------
r230041 | file | 2009-11-13 13:51:44 -0600 (Fri, 13 Nov 2009) | 23 lines

Merged revisions 230039 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r230039 | file | 2009-11-13 13:44:53 -0600 (Fri, 13 Nov 2009) | 16 lines
 
 Merged revisions 230038 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r230038 | file | 2009-11-13 13:44:07 -0600 (Fri, 13 Nov 2009) | 9 lines
   
   Fix a crash caused by two threads thinking they should both free the
   chan_local private structure when only one should.
   
   (closes issue ASTERISK-14304)
   Reported by: sroberts
   Patches:
         Issue15314_Move_Nulling_owner.patch uploaded by davidw (license 780)
   Tested by: davidw, lottc
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=230041
By: Digium Subversion (svnbot) 2009-11-13 13:52:00.000-0600

Repository: asterisk
Revision: 230042

_U  branches/1.6.2/
U   branches/1.6.2/channels/chan_local.c

------------------------------------------------------------------------
r230042 | file | 2009-11-13 13:52:00 -0600 (Fri, 13 Nov 2009) | 23 lines

Merged revisions 230039 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
 r230039 | file | 2009-11-13 13:44:53 -0600 (Fri, 13 Nov 2009) | 16 lines
 
 Merged revisions 230038 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r230038 | file | 2009-11-13 13:44:07 -0600 (Fri, 13 Nov 2009) | 9 lines
   
   Fix a crash caused by two threads thinking they should both free the
   chan_local private structure when only one should.
   
   (closes issue ASTERISK-14304)
   Reported by: sroberts
   Patches:
         Issue15314_Move_Nulling_owner.patch uploaded by davidw (license 780)
   Tested by: davidw, lottc
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=230042