[Home]

Summary:ASTERISK-21763: asterisk -r Bus Error on Debian/sparc
Reporter:Jeremy Kister (jkister)Labels:
Date Opened:2013-05-07 13:36:42Date Closed:2013-08-31 00:15:10
Priority:MajorRegression?Yes
Status:Closed/CompleteComponents:General
Versions:1.8.20.0 1.8.21.0 10.12.2 11.4.0 Frequency of
Occurrence		Constant
Related
Issues:
is related toASTERISK-21665 11.X Crash on debian/sparc with SIGBUS, Bus Error
Environment:Debian 6 SPARCAttachments:( 0) astmm_align.patch
( 1) backtrace1.txt
( 2) break-main-backtrace.txt
( 3) diff-uNrw-between-19-and-20.diff
( 4) pimped-backtrace1.txt
Description:as root:
# /etc/init.d/asterisk stop
stopping asterisk.
Asterisk ended with exit status 0
Asterisk shutdown normally.
# /etc/init.d/asterisk start
starting asterisk.
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
Bus error (core dumped)

as non-root:
# asterisk -r
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
Bus error

as root:
# asterisk -r
Asterisk 10.12.2, Copyright (C) 1999 - 2012 Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Connected to Asterisk 10.12.2 currently running on pbx1 (pid = 8635)
Verbosity is at least 3
pbx1*CLI>
Comments:By: Jeremy Kister (jkister) 2013-05-07 13:38:26.399-0500

Related To: ASTERISK-21665
By: Jeremy Kister (jkister) 2013-05-07 13:42:06.819-0500

this is different from ASTERISK-21665, because asterisk starts and behaves nicely.  but something (possibly rasterisk) is core dumping right after starting.
By: Jeremy Kister (jkister) 2013-05-07 13:45:25.731-0500

[link to backtrace1.txt|https://issues.asterisk.org/jira/secure/attachment/47301/backtrace1.txt]

{noformat}
> id
uid=6447(jk) gid=14(super-users) groups=14(super-users),0(root),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev)
> sudo rasterisk asterisk -nrx 'core show version'
Asterisk 10.12.2 built by root @ pbx1.offphl on a sparc64 running Linux on 2013-04-26 19:05:40 UTC
> rasterisk asterisk -nrx 'core show version'
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
Bus error
{noformat}


By: Rusty Newton (rnewton) 2013-05-08 18:22:40.626-0500

Attaching Jeremy's backtrace
By: Rusty Newton (rnewton) 2013-05-08 18:25:47.120-0500

Have you been able to reproduce either of these issues with Debian on a x86/x86_64 non-sparc system?
By: Jeremy Kister (jkister) 2013-05-09 14:01:52.481-0500

No, all the asterisk flavors with debian on x86/x86_64 work just fine.

By: Jeremy Kister (jkister) 2013-05-09 17:32:37.728-0500

The Bus Error was introduced on 1.8.20.0-rc1 - no bus error on 1.8.19.0.

so that narrows it down to ~2000 lines in 49 files :D
By: Jeremy Kister (jkister) 2013-05-09 17:59:04.618-0500

here is a backtrace for the "asterisk -nrx 'core show version'" Bus Error on 1.8.20.0-rc1
By: Walter Doekes (wdoekes) 2013-05-10 04:12:46.912-0500

So.. it should be something in the atexit handler. Since the message is printed, and it does nothing more than exit(1) there.
By: Walter Doekes (wdoekes) 2013-05-10 04:13:20.811-0500

Thanks for the detailed digging by the way :)
By: Walter Doekes (wdoekes) 2013-05-10 04:23:20.652-0500

Are you running with MALLOC_DEBUG ?
By: Walter Doekes (wdoekes) 2013-05-10 05:22:37.020-0500

If you are running MALLOC_DEBUG {{astmm_align.patch}} might do something for you.

If you aren't, then I'm still unsure where it might be. There aren't many atexit handlers called. (Really only {{mm_atexit_final}}.)

{noformat}
gdb asterisk
break main
run -r
next
(hold enter to repeat)
{noformat}

(There are undoubtedly better gdb ways of doing the above.)
By: Jeremy Kister (jkister) 2013-05-10 08:59:11.223-0500

no MALLOC_DEBUG - just DONT_OPTIMIZE and DEBUG_THREADS

By: Jeremy Kister (jkister) 2013-05-10 09:10:05.135-0500

you were right; we bus error after exit(1)
By: Jeremy Kister (jkister) 2013-05-10 09:36:19.493-0500

i applied [^astmm_align.patch] (for giggles).  it didnt solve the bus error.
By: Jeremy Kister (jkister) 2013-06-16 22:28:43.276-0500

not sure if it's interesting, but if i:
{noformat}
cp asterisk-1.8.19.1/main/asterisk.c asterisk-1.8.20.0-rc1/main
cd asterisk-1.8.20.0-rc1
make install
{noformat}

the bus error does not occur.
By: Walter Doekes (wdoekes) 2013-06-17 02:33:52.678-0500

> not sure if it's interesting

I assume you mean asterisk.c.


I'm sorry that I haven't had the time to investigate further.

But the one time I tested things on your box, I tried to tackle the 'asterisk -r' as non-root bus error first. If my memory serves me right, the bus error disappeared if I removed this line (from asterisk.c):
{noformat}
+ fprintf(stderr, "Unable to connect to remote asterisk (does %s exist?)\n", ast_config_AST_SOCKET);
{noformat}

That felt totally random.

Supposedly there is some call along the line that messes up the stderr FILE internals. Or, the stderr FILE uses a misaligned mutex itself...


By: Jeremy Kister (jkister) 2013-06-17 02:52:26.137-0500

bah. asterisk.c, yes.

no apologies necessary!  glad someone else's eyes are on it.

By: Walter Doekes (wdoekes) 2013-06-19 09:19:30.272-0500

A-ha.

Before:

{noformat}
wdoekes@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1$ asterisk -r
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
Bus error
wdoekes@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1$
{noformat}

After:

{noformat}
wdoekes@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1$ ./main/asterisk -r
Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)
wdoekes@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1$
{noformat}

This is the one: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261

{noformat}
gcc  -o asterisk -Wl,--export-dynamic -Wl,--version-script,asterisk.exports -Wl,--dynamic-list,asterisk.dynamics      abstract_jb.o acl.o ad ...
{noformat}

{noformat}
--- main/asterisk.exports.orig  2013-06-19 10:17:48.000000000 -0400
+++ main/asterisk.exports       2013-06-19 10:06:39.000000000 -0400
@@ -46,6 +46,7 @@
               res_srtp;
               res_srtp_policy;
               secure_call_info;
+               _IO_stdin_used;
       local:
               *;
};
{noformat}

That's one sigbus down..
By: Walter Doekes (wdoekes) 2013-06-19 09:22:37.219-0500

For the record:
{noformat}
root@t1000:/usr/local/src/sparcbus# cat Makefile
.PHONY: all

all: foo foo2 foo3

foo: foo.c
       gcc -o foo foo.c
foo2: ver2 foo.c
       gcc -o foo2 foo.c -Wl,--version-script,ver2
foo3: ver3 foo.c
       gcc -o foo3 foo.c -Wl,--version-script,ver3

define ver2_body
{
       local: *;
};
endef
export ver2_body

ver2: Makefile
       sh -c 'echo "$$ver2_body" >ver2'

define ver3_body
{
       global:
               _IO_stdin_used;
       local:
               *;
};
endef
export ver3_body

ver3: Makefile
       sh -c 'echo "$$ver3_body" >ver3'
{noformat}

{noformat}
root@t1000:/usr/local/src/sparcbus# cat foo.c
#include <stdio.h>
#include <stdlib.h>

int main() {
 printf("stdout=%p &_IO_2_1_stdout_=%p\n", stdout, &_IO_2_1_stdout_);
 setbuf(stdout, 0);
 return 0;
}
{noformat}

{noformat}
root@t1000:/usr/local/src/sparcbus# make
gcc -o foo foo.c
gcc -o foo2 foo.c -Wl,--version-script,ver2
gcc -o foo3 foo.c -Wl,--version-script,ver3

root@t1000:/usr/local/src/sparcbus# ./foo
stdout=0x207e0 &_IO_2_1_stdout_=0x207e0

root@t1000:/usr/local/src/sparcbus# ./foo2
stdout=0xf787f2fc &_IO_2_1_stdout_=0x207c0
Bus error

root@t1000:/usr/local/src/sparcbus# ./foo3
stdout=0x207e0 &_IO_2_1_stdout_=0x207e0
{noformat}
By: Jeremy Kister (jkister) 2013-06-19 21:52:20.108-0500

I can confirm your fix on 11.5.0 and 1.8.22.0 - yay!
By: Jeremy Kister (jkister) 2013-08-31 00:15:10.402-0500

https://reviewboard.asterisk.org/r/2760/