ASTERISK-10573: Crash at random through the day. The box is Sip+H323 only

[Home]

Summary: ASTERISK-10573: Crash at random through the day. The box is Sip+H323 only

Reporter: Private Name (falves11) Labels:

Date Opened: 2007-10-18 19:24:35 Date Closed: 2008-02-24 06:17:20.000-0600

Priority: Major Regression? No

Status: Closed/Complete Components: Channels/General

Versions: Frequency of
Occurrence

Related
Issues:

Environment: Attachments: ( 0) bt_full4.txt
( 1) bt_full5.txt
( 2) bt_full6.txt
( 3) crash_1.txt

Description: This happens in a production server at random. The code is SVN trunk

****** ADDITIONAL INFORMATION ******

#0 0x080847d6 in ast_waitfor_nandfds_complex (c=0xb5480d70, n=2, ms=0xb5481cf8) at channel.c:1921
1921 ast_clear_flag(winner, AST_FLAG_EXCEPTION);
(gdb) bt full
#0 0x080847d6 in ast_waitfor_nandfds_complex (c=0xb5480d70, n=2, ms=0xb5481cf8) at channel.c:1921
__p = 16
__x = 0
aed = (struct ast_epoll_data *) 0x893d200
start = {tv_sec = 1192747301, tv_usec = 950168}
res = 1
i = 0
ev = {{events = 1, data = {ptr = 0x893d200, fd = 143905280, u32 = 143905280, u64 = 6605803606528}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0,
u64 = 0}} <repeats 24 times>}
whentohangup = 0
diff = -1253569592
rms = 89998
now = -1253569592
winner = (struct ast_channel *) 0x4
__PRETTY_FUNCTION__ = "ast_waitfor_nandfds_complex"
#1 0x08084955 in ast_waitfor_nandfds (c=0xb5480d70, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0xb5481cf8) at channel.c:1949
No locals.
#2 0x0808499c in ast_waitfor_n (c=0xb5480d70, n=2, ms=0xb5481cf8) at channel.c:1955
No locals.
#3 0xb6b2028b in wait_for_answer (in=0x841ca40, outgoing=0xb3885cd8, to=0xb5481cf8, peerflags=0xb5481d48, pa=0xb5481510, num_in=0xb5481ce0,
result=0xb5481500) at app_dial.c:614
o = (struct chanlist *) 0x0
pos = 2
numlines = 1
winner = (struct ast_channel *) 0x841c500
watchers = {0x841ca40, 0xb380e578, 0x0 <repeats 22 times>, 0xb71954e4, 0xb71954e4, 0x0, 0xb71954e4, 0xb5480e30, 0xb5480e0c, 0xb725dff4, 0xb5480e3c,
0xb5480e18, 0xb7196901, 0xb5480e3c, 0xb3886a99, 0xb3886bcb, 0x0, 0xb3886bcb, 0xb725dff4, 0xb6942fc2, 0xb71a0b61, 0x5f480f28, 0xb6b2d29c, 0xab227498,
0x5b1700, 0xb5480e68, 0x806c796, 0x2e, 0x841cc18, 0xb5480e3c, 0xfbad8001, 0xb3886a99, 0x0, 0x820a828, 0x820a828, 0x370, 0x0, 0xb3886a99, 0xb6b2d29c,
0xb380e578, 0xb3885cd8, 0xb5480e88, 0x80c67b0, 0x0, 0x0, 0x2, 0xb3886a5f, 0x81eec80, 0x0, 0xb5480ef8, 0x80c6a64, 0x81ee2b0, 0x81ee310, 0x2, 0x0, 0x0,
0xffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb725dc00, 0xb725da20, 0x0, 0xb725dff4, 0xb725da20, 0xb3885cd8, 0xb6b2d29c, 0x80c60b7, 0x816f9c0, 0x0,
0xb5480ef8, 0x80d55d1, 0x816f9c0, 0xb6b2d29c, 0xb5481168, 0x80c9997, 0x841ca40, 0x0, 0xb5480f30, 0x841cbc8, 0x841cc18, 0xffffffff, 0x0, 0x815543d, 0x2,
0xb3885cd8, 0xb5480f58, 0x810c0ac, 0x81ee304, 0x0 <repeats 70 times>, 0xb71986a4, 0x0 <repeats 30 times>, 0xb7197d44, 0x0, 0x0, 0x0, 0x0, 0x0, 0x50, 0x0,
0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb725dff4, 0xb3800010, 0xb5481158, 0x807dd6e, 0xb389aeaa, 0xb387df72, 0x46, 0xb725dff4, 0x1, 0xb6b2a740, 0x0, 0xb387df68,
0x0, 0xb6b2a740, 0x1, 0x0, 0xb387df68, 0x0, 0xb387df68, 0xb7161cfb, 0x1, 0xb387df68, 0xb5481178, 0x807ddb1, 0xb300ae50, 0xb380e578, 0xb724a3c0, 0x0}
num = {chan = 0x841ca40, busy = 0, congestion = 0, nochan = 0}
prestart = 0
orig = 90000
peer = (struct ast_channel *) 0x0
single = 1
epollo = (struct chanlist *) 0x0
__PRETTY_FUNCTION__ = "wait_for_answer"
#4 0xb6b25295 in dial_exec_full (chan=0x841ca40, data=0xb5485f70, peerflags=0xb5481d48, continue_exec=0x0) at app_dial.c:1492
res = 0
rest = 0x0
cur = 0x0
outgoing = (struct chanlist *) 0xb3885cd8
peer = (struct ast_channel *) 0x841ca40
to = 89998
num = {chan = 0x841ca40, busy = 0, congestion = 0, nochan = 0}
cause = 0
numsubst = "14156952220@66.28.147.100\000Hµ9z\v\bhÎ\026\b\000\000\000\000\b\034Hµ¯_\v\b°â\211³\001\000\000\000\001\000\000\000ÈËA\b@ÊA\b°â\211³h\034Hµ¤\206\031·\220A\036\b°â\211³9\034Hµ\036\035Hµ@ÊA\bP\034Hµê\034HµÅ\206\f\bCALLERID\000all\000\000\026·h323\0001415695RD\r\b\000\000\000\000°â\211³9\034Hµ\200Á}¶\000\000\000\0000\034Hµè\034Hµ¯D\r\b@ÊA\b \034Hµ\036\035Hµ\231\231\231\031\000\000\000\000ôß%·"...
cidname = '\0' <repeats 79 times>
config = {features_caller = {flags = 0}, features_callee = {flags = 0}, start_time = {tv_sec = 0, tv_usec = 0}, feature_timer = 0, timelimit = 0,
play_warning = 0, warning_freq = 0, warning_sound = 0xb6b2ad9e "timeleft", end_sound = 0x0, start_sound = 0x0, firstpass = 0, flags = 0}
calldurationlimit = 3600
dtmfcalled = 0x0
dtmfcalling = 0x0
pa = {sentringing = 0, privdb_val = 0, privcid = '\0' <repeats 255 times>, privintro = '\0' <repeats 1023 times>,
---Type <return> to continue, or q <return> to quit---
status = "NOANSWER\000GS", '\0' <repeats 244 times>}
sentringing = 0
moh = 0
outbound_group = 0x0
result = 0
start_time = 1192747299
parse = 0xb5481210 "SIP"
opermode = 0
args = {argc = 3, argv = 0xb54814d4, peers = 0xb5481210 "SIP", timeout = 0xb548122e "90", options = 0xb5481231 "L(3600000", url = 0x0}
opts = {flags = 512}
opt_args = {0x0, 0x0, 0x0, 0xb5481233 "3600000", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
__PRETTY_FUNCTION__ = "dial_exec_full"
ASTERISK-1 0xb6b270e3 in dial_exec (chan=0x841ca40, data=0xb5485f70) at app_dial.c:1846
peerflags = {flags = 0}
ASTERISK-2 0x080c600b in pbx_exec (c=0x841ca40, app=0x81c5de8, data=0xb5485f70) at pbx.c:596
res = 138529344
u = (struct ast_module_user *) 0xb389e2b0
saved_c_appl = 0x0
saved_c_data = 0x0
ASTERISK-3 0x080c9619 in pbx_extension_helper (c=0x841ca40, con=0x0, context=0x841cbc8 "h323", exten=0x841cc18 "14156952220", priority=12, label=0x0,
callerid=0xb38b2328 "2014532452", action=E_SPAWN) at pbx.c:1906
e = (struct ast_exten *) 0x81ee810
app = (struct ast_app *) 0x81c5de8
res = 8
q = {incstack = {0x0 <repeats 128 times>}, stacklen = 0, status = 5, swo = 0x0, data = 0x0, foundcontext = 0x841cbc8 "h323"}
passdata = "SIP/14156952220@66.28.147.100,90,L(3600000)", '\0' <repeats 8148 times>
matching_action = 0
__PRETTY_FUNCTION__ = "pbx_extension_helper"
ASTERISK-4 0x080ca92b in ast_spawn_extension (c=0x841ca40, context=0x841cbc8 "h323", exten=0x841cc18 "14156952220", priority=12, callerid=0xb38b2328 "2014532452")
at pbx.c:2397
No locals.
ASTERISK-5 0x080cae7d in __ast_pbx_run (c=0x841ca40) at pbx.c:2496
dst_exten = "\020\203HµL\203Hµ\000\000\000\000X\203Hµ\030\203Hµ\034\203Hµ\\\203Hµ", '\0' <repeats 36 times>, "`\203HµtÁ\f\b\000\000\000\000\000\000\000\000ô/ë·0 ê·\000\000\000\0004\000 ¶8\000 ¶\020\000 ¶\f\203Hµ\000\000\000\000\020\203Hµ\024\203Hµ\000\000\000\000 \203Hµ\030\203HµHø%·D}\031·\020\000\000\000\000\000\000\000°s\031·\000\000\000\000\f\000\000\000ôß%·\f\000\000\000@ø%·\b\203HµV¤\031·@ø%·\f\000\000\000\000\000\000\000x\t\002\000\f\000\000\000\210\206Ø\bô/ë·\000\000\000\000\000\000\000\0008\203Hµ¹£\020\b\001\000"...
pos = 0
digit = 0
found = 1
res = 0
autoloopflag = 0
error = 0
__PRETTY_FUNCTION__ = "__ast_pbx_run"
ASTERISK-6 0x080cc163 in pbx_thread (data=0x841ca40) at pbx.c:2752
c = (struct ast_channel *) 0x841ca40
ASTERISK-7 0x0810b42f in dummy_start (data=0x841c4f8) at utils.c:807
_buffer = {__routine = 0x806ba7d <ast_unregister_thread>, __arg = 0xb5488ba0, __canceltype = -1253538872, __prev = 0x0}
ret = (void *) 0xb725c6f4
a = {start_routine = 0x80cc14c <pbx_thread>, data = 0x841ca40,
name = 0x841c650 "pbx_thread", ' ' <repeats 11 times>, "started at [ 2773] pbx.c ast_pbx_start()"}
ASTERISK-8 0xb7ea93cc in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
ASTERISK-9 0xb71fec3e in clone () from /lib/tls/libc.so.6
No symbol table info available.
(gdb)

Comments: By: Joshua C. Colp (jcolp) 2007-10-19 09:26:12

Please attach backtraces as files next time. As for this one please open the backtrace, type the following, and add the results to this bug:

frame 0
print winner
print *winner
By: Andrey S Pankov (casper) 2007-10-22 08:11:33

From the post:

winner = (struct ast_channel *) 0x4

That should be enough I think....
By: Private Name (falves11) 2007-10-22 09:30:09

Dear Casper
you wrote:
"winner = (struct ast_channel *) 0x4

That should be enough I think.... "

Can you please elaborate? Is there an obvious patch in the code that i can make and would make it more stable?
By: Digium Subversion (svnbot) 2008-02-21 10:41:35.000-0600

Repository: asterisk
Revision: 104019

U trunk/configure
U trunk/configure.ac
U trunk/include/asterisk/autoconfig.h.in

------------------------------------------------------------------------
r104019 | file | 2008-02-21 10:41:31 -0600 (Thu, 21 Feb 2008) | 8 lines

Disable epoll as it has caused more obscure issues then any of my previous code. I will continue to work on it in a separate branch to make it stable for a release and test it against the following issues.
(closes issue ASTERISK-10770)
Reported by: falves11
(closes issue ASTERISK-11131)
Reported by: davevg
(closes issue ASTERISK-10573)
Reported by: falves11

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=104019
By: Digium Subversion (svnbot) 2008-02-23 21:17:45.000-0600

Repository: asterisk
Revision: 104076

_U team/seanbright/NoLossCDR-Redux/
U team/seanbright/NoLossCDR-Redux/CHANGES
U team/seanbright/NoLossCDR-Redux/UPGRADE.txt
U team/seanbright/NoLossCDR-Redux/channels/chan_sip.c
U team/seanbright/NoLossCDR-Redux/channels/chan_zap.c
U team/seanbright/NoLossCDR-Redux/configure
U team/seanbright/NoLossCDR-Redux/configure.ac
U team/seanbright/NoLossCDR-Redux/doc/manager_1_1.txt
U team/seanbright/NoLossCDR-Redux/include/asterisk/autoconfig.h.in
U team/seanbright/NoLossCDR-Redux/include/asterisk/manager.h
U team/seanbright/NoLossCDR-Redux/main/manager.c
U team/seanbright/NoLossCDR-Redux/res/res_agi.c
U team/seanbright/NoLossCDR-Redux/res/res_config_pgsql.c
U team/seanbright/NoLossCDR-Redux/utils/astman.c

------------------------------------------------------------------------
r104076 | seanbright | 2008-02-23 21:17:26 -0600 (Sat, 23 Feb 2008) | 142 lines

Merged revisions 104014,104016,104019-104020,104024-104025,104028-104029,104031,104036,104038-104039,104045,104073-104074 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r104014 | tilghman | 2008-02-21 00:21:39 -0500 (Thu, 21 Feb 2008) | 6 lines

Ignore some more unused generated events.
(closes issue ASTERISK-11487)
Reported by: junky
Patches:
astman_events.diff uploaded by junky (license 177)

................
r104016 | kpfleming | 2008-02-21 09:44:04 -0500 (Thu, 21 Feb 2008) | 10 lines

Merged revisions 104015 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r104015 | kpfleming | 2008-02-21 08:33:51 -0600 (Thu, 21 Feb 2008) | 2 lines

reduce the likelihood that HTTP Manager session ids will consist of primarily '1' bits

........

................
r104019 | file | 2008-02-21 11:44:57 -0500 (Thu, 21 Feb 2008) | 8 lines

Disable epoll as it has caused more obscure issues then any of my previous code. I will continue to work on it in a separate branch to make it stable for a release and test it against the following issues.
(closes issue ASTERISK-10770)
Reported by: falves11
(closes issue ASTERISK-11131)
Reported by: davevg
(closes issue ASTERISK-10573)
Reported by: falves11

................
r104020 | mmichelson | 2008-02-21 11:46:37 -0500 (Thu, 21 Feb 2008) | 7 lines

Don't print the fact that we are using dead mode in AGI if called from the
'h' extension since it is well-known that it will be running in dead mode.

(closes issue ASTERISK-11491)
Reported by: explidous

................
r104024 | dbailey | 2008-02-21 12:38:40 -0500 (Thu, 21 Feb 2008) | 4 lines

Added configuration distinction between neon and fsk mwi detection
Add the detection for neon MWI events
got rid of extraneous handle_init_event call in monitor loop

................
r104025 | mmichelson | 2008-02-21 12:44:34 -0500 (Thu, 21 Feb 2008) | 4 lines

Instead of a notice, make the message about a hung-up channel a debug message, and revert the original
logic on the if statement. Thanks to Juggie for bringing this to my attention.

................
r104028 | mmichelson | 2008-02-21 16:09:11 -0500 (Thu, 21 Feb 2008) | 14 lines

Blocked revisions 104026 via svnmerge

........
r104026 | mmichelson | 2008-02-21 14:12:38 -0600 (Thu, 21 Feb 2008) | 7 lines

Remove an incorrect debug message. It reported that it had received a specific event and tried to report
which event was received. What actually was happening was that it was reporting the number of bytes returned
from a call to read().

Thanks to Jared Smith for bringing the issue up on IRC

........

................
r104029 | mmichelson | 2008-02-21 16:09:54 -0500 (Thu, 21 Feb 2008) | 11 lines

Blocked revisions 104027 via svnmerge

........
r104027 | mmichelson | 2008-02-21 15:05:42 -0600 (Thu, 21 Feb 2008) | 4 lines

And as a followup to revision 104026, completely remove event-related
calls from a section of code where we know there was no event to handle or get.

........

................
r104031 | russell | 2008-02-21 16:27:24 -0500 (Thu, 21 Feb 2008) | 1 line

fix a typo
................
r104036 | tilghman | 2008-02-22 17:39:21 -0500 (Fri, 22 Feb 2008) | 7 lines

Allow database password to be NULL and several other cleanups.
(closes issue ASTERISK-11493)
Reported by: bukaj
Patches:
20080222__bug12048.diff.txt uploaded by Corydon76 (license 14)
Tested by: bukaj

................
r104038 | tilghman | 2008-02-22 17:48:18 -0500 (Fri, 22 Feb 2008) | 14 lines

Merged revisions 104037 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r104037 | tilghman | 2008-02-22 16:45:14 -0600 (Fri, 22 Feb 2008) | 6 lines

Backwards debug message.
(closes issue ASTERISK-11496)
Reported by: flefoll
Patches:
chan_sip.c.br14.patch_found-notfound uploaded by flefoll (license 244)

........

................
r104039 | tilghman | 2008-02-22 17:55:35 -0500 (Fri, 22 Feb 2008) | 2 lines

Move Originate to a separate privilege and require the additional System privilege to call out to a subshell.

................
r104045 | dbailey | 2008-02-22 18:56:55 -0500 (Fri, 22 Feb 2008) | 2 lines

Add protection to chan_zap build when NEONMWI events are not defined

................
r104073 | murf | 2008-02-23 19:44:14 -0500 (Sat, 23 Feb 2008) | 1 line

On a 64-bit machine, with dev-mode turned on, and pgsql installed, I get warnings that stops the compile. They are fixed now.
................
r104074 | murf | 2008-02-23 21:37:08 -0500 (Sat, 23 Feb 2008) | 1 line

Enforce a space between function args as per code review.
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=104076
By: Digium Subversion (svnbot) 2008-02-24 06:17:20.000-0600

Repository: asterisk
Revision: 104077

_U team/group/multiparking/
U team/group/multiparking/CHANGES
U team/group/multiparking/UPGRADE.txt
U team/group/multiparking/channels/chan_sip.c
U team/group/multiparking/channels/chan_zap.c
U team/group/multiparking/configure
U team/group/multiparking/configure.ac
U team/group/multiparking/doc/manager_1_1.txt
U team/group/multiparking/include/asterisk/autoconfig.h.in
U team/group/multiparking/include/asterisk/manager.h
U team/group/multiparking/main/manager.c
U team/group/multiparking/res/res_agi.c
U team/group/multiparking/res/res_config_pgsql.c
U team/group/multiparking/utils/astman.c

------------------------------------------------------------------------
r104077 | mvanbaak | 2008-02-24 06:17:17 -0600 (Sun, 24 Feb 2008) | 142 lines

Merged revisions 104014,104016,104019-104020,104024-104025,104028-104029,104031,104036,104038-104039,104045,104073-104074 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk

................
r104014 | tilghman | 2008-02-21 06:21:39 +0100 (Thu, 21 Feb 2008) | 6 lines

Ignore some more unused generated events.
(closes issue ASTERISK-11487)
Reported by: junky
Patches:
astman_events.diff uploaded by junky (license 177)

................
r104016 | kpfleming | 2008-02-21 15:44:04 +0100 (Thu, 21 Feb 2008) | 10 lines

Merged revisions 104015 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r104015 | kpfleming | 2008-02-21 08:33:51 -0600 (Thu, 21 Feb 2008) | 2 lines

reduce the likelihood that HTTP Manager session ids will consist of primarily '1' bits

........

................
r104019 | file | 2008-02-21 17:44:57 +0100 (Thu, 21 Feb 2008) | 8 lines

Disable epoll as it has caused more obscure issues then any of my previous code. I will continue to work on it in a separate branch to make it stable for a release and test it against the following issues.
(closes issue ASTERISK-10770)
Reported by: falves11
(closes issue ASTERISK-11131)
Reported by: davevg
(closes issue ASTERISK-10573)
Reported by: falves11

................
r104020 | mmichelson | 2008-02-21 17:46:37 +0100 (Thu, 21 Feb 2008) | 7 lines

Don't print the fact that we are using dead mode in AGI if called from the
'h' extension since it is well-known that it will be running in dead mode.

(closes issue ASTERISK-11491)
Reported by: explidous

................
r104024 | dbailey | 2008-02-21 18:38:40 +0100 (Thu, 21 Feb 2008) | 4 lines

Added configuration distinction between neon and fsk mwi detection
Add the detection for neon MWI events
got rid of extraneous handle_init_event call in monitor loop

................
r104025 | mmichelson | 2008-02-21 18:44:34 +0100 (Thu, 21 Feb 2008) | 4 lines

Instead of a notice, make the message about a hung-up channel a debug message, and revert the original
logic on the if statement. Thanks to Juggie for bringing this to my attention.

................
r104028 | mmichelson | 2008-02-21 22:09:11 +0100 (Thu, 21 Feb 2008) | 14 lines

Blocked revisions 104026 via svnmerge

........
r104026 | mmichelson | 2008-02-21 14:12:38 -0600 (Thu, 21 Feb 2008) | 7 lines

Remove an incorrect debug message. It reported that it had received a specific event and tried to report
which event was received. What actually was happening was that it was reporting the number of bytes returned
from a call to read().

Thanks to Jared Smith for bringing the issue up on IRC

........

................
r104029 | mmichelson | 2008-02-21 22:09:54 +0100 (Thu, 21 Feb 2008) | 11 lines

Blocked revisions 104027 via svnmerge

........
r104027 | mmichelson | 2008-02-21 15:05:42 -0600 (Thu, 21 Feb 2008) | 4 lines

And as a followup to revision 104026, completely remove event-related
calls from a section of code where we know there was no event to handle or get.

........

................
r104031 | russell | 2008-02-21 22:27:24 +0100 (Thu, 21 Feb 2008) | 1 line

fix a typo
................
r104036 | tilghman | 2008-02-22 23:39:21 +0100 (Fri, 22 Feb 2008) | 7 lines

Allow database password to be NULL and several other cleanups.
(closes issue ASTERISK-11493)
Reported by: bukaj
Patches:
20080222__bug12048.diff.txt uploaded by Corydon76 (license 14)
Tested by: bukaj

................
r104038 | tilghman | 2008-02-22 23:48:18 +0100 (Fri, 22 Feb 2008) | 14 lines

Merged revisions 104037 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
r104037 | tilghman | 2008-02-22 16:45:14 -0600 (Fri, 22 Feb 2008) | 6 lines

Backwards debug message.
(closes issue ASTERISK-11496)
Reported by: flefoll
Patches:
chan_sip.c.br14.patch_found-notfound uploaded by flefoll (license 244)

........

................
r104039 | tilghman | 2008-02-22 23:55:35 +0100 (Fri, 22 Feb 2008) | 2 lines

Move Originate to a separate privilege and require the additional System privilege to call out to a subshell.

................
r104045 | dbailey | 2008-02-23 00:56:55 +0100 (Sat, 23 Feb 2008) | 2 lines

Add protection to chan_zap build when NEONMWI events are not defined

................
r104073 | murf | 2008-02-24 01:44:14 +0100 (Sun, 24 Feb 2008) | 1 line

On a 64-bit machine, with dev-mode turned on, and pgsql installed, I get warnings that stops the compile. They are fixed now.
................
r104074 | murf | 2008-02-24 03:37:08 +0100 (Sun, 24 Feb 2008) | 1 line

Enforce a space between function args as per code review.
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=104077