| Summary: | ASTERISK-21665: 11.X Crash on debian/sparc with SIGBUS, Bus Error | ||||||
| Reporter: | Jeremy Kister (jkister) | Labels: | |||||
| Date Opened: | 2013-04-22 13:21:27 | Date Closed: | 2016-02-12 08:21:20.000-0600 | ||||
| Priority: | Critical | Regression? | |||||
| Status: | Closed/Complete | Components: | Core/General | ||||
| Versions: | 11.3.0 11.4.0 11.5.0 | Frequency of Occurrence | Constant | ||||
| Related Issues: |
| ||||||
| Environment: | Debian 6 & 7, sparc64 | Attachments: | ( 0) core_stop_now_backtrace.txt ( 1) modules.conf ( 2) sparc_alignment_issues_2.diff ( 3) sparc_alignment_issues_3.diff ( 4) sparc_alignment_issues.diff | ||||
| Description: | Asterisk 11.4.0-rc1 immediately dumps core with error "Bus Error" | ||||||
| Comments: | By: Jeremy Kister (jkister) 2013-04-22 13:24:34.770-0500 i have compiled with DONT_OPTIMIZE and DEBUG_THREADS, but this backtrace is not behaving how i expect- {noformat} gdb asterisk core.pbx1-2013-04-22T14\:15\:56-0400 GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. Reading symbols from /usr/lib/libasteriskssl.so.1...done. Loaded symbols for /usr/lib/libasteriskssl.so.1 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /usr/lib/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsqlite3.so.0 Reading symbols from /usr/lib/v9/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/v9/libssl.so.0.9.8 Reading symbols from /usr/lib/v9/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/v9/libcrypto.so.0.9.8 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libncurses.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libncurses.so.5 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Core was generated by `/usr/sbin/asterisk -f -vvvg -c'. Program terminated with signal 10, Bus error. #0 0xf79ff9f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 (gdb) bt #0 0xf79ff9f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0xf7f06abc in pthread_cond_init () from /lib/libc.so.6 Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) thread apply all bt Thread 1 (Thread 2191): #0 0xf79ff9f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0xf7f06abc in pthread_cond_init () from /lib/libc.so.6 Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) {noformat} By: Jeremy Kister (jkister) 2013-04-22 13:44:11.889-0500 modules.conf seems irrelevant - i commented out /every/ load (and preload) and Asterisk still dumps core. By: Jeremy Kister (jkister) 2013-04-22 14:06:00.166-0500 1.8.19.0 works fine By: Walter Doekes (wdoekes) 2013-04-23 03:58:12.729-0500 What happens if you start it from gdb directly? {noformat} gdb `which asterisk` run -c {noformat} If that doesn't yield more info.. valgrind? By: Jeremy Kister (jkister) 2013-04-23 10:01:44.420-0500 {noformat} pbx1.offphl> gdb `which asterisk` GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. (gdb) run -c Starting program: /usr/sbin/asterisk -c [Thread debugging using libthread_db enabled] Asterisk 11.4.0-rc1, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Program received signal SIGBUS, Bus error. 0xf7a279f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 {noformat} I found http://svn.ohnosec.org/svn/projects/allstar/astsrc-1.4.19/trunk/asterisk/doc/valgrind.txt about valgrind - i'll try that. By: Walter Doekes (wdoekes) 2013-04-23 10:30:22.808-0500 And if you set a breakpoint on pthread_cond_init? {noformat} break pthread_cond_init run -c bt cont bt cont {noformat} That might you a backtrace before the stack gets corrupted. By: Jeremy Kister (jkister) 2013-04-23 10:36:43.224-0500 {noformat} # gdb `which asterisk` GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. (gdb) break pthread_cond_init Breakpoint 1 at 0x27ef18 (gdb) run -c Starting program: /usr/sbin/asterisk -c [Thread debugging using libthread_db enabled] Asterisk 11.4.0-rc1, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Program received signal SIGBUS, Bus error. 0xf7a279f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 (gdb) bt #0 0xf7a279f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0xf7f2eabc in pthread_cond_init () from /lib/libc.so.6 #2 0x0014d344 in __ast_cond_init (filename=0x264db8 "taskprocessor.c", lineno=455, func=0x265458 "ast_taskprocessor_get", cond_name=0x2652c8 "&p->poll_cond", cond=0x2e40c4, cond_attr=0x0) at lock.c:454 #3 0x001ee750 in ast_taskprocessor_get (name=0x2453b0 "core_event_dispatcher", create=TPS_REF_DEFAULT) at taskprocessor.c:455 #4 0x001032bc in ast_event_init () at event.c:1873 #5 0x00066028 in main (argc=2, argv=0xffffd974) at asterisk.c:4008 (gdb) cont Continuing. Program terminated with signal SIGBUS, Bus error. The program no longer exists. (gdb) bt No stack. (gdb) cont The program is not being run. (gdb) {noformat} By: Walter Doekes (wdoekes) 2013-04-23 14:25:53.277-0500 I don't know about valgrind on sparc, nor do I know whether it gives us more info. But the gdb output was helpful. Could you try the attached patch? It may do nothing. It may allow gdb to cont(inue) once before SIGBUGsing. By: Jeremy Kister (jkister) 2013-04-23 14:38:04.997-0500 patch did not apply clean to 11.4.0-rc1, but was easy to do manually. did not seem to make a difference, thank you for your support btw. {noformat} > gdb `which asterisk` GNU gdb (GDB) 7.0.1-debian [...] (gdb) run -c [...] Program received signal SIGBUS, Bus error. 0xf7b8f9f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8f9f8 in pthread_cond_init@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0xf7f2eabc in pthread_cond_init () from /lib/libc.so.6 #2 0x00154464 in __ast_cond_init (filename=0x2659b8 "taskprocessor.c", lineno=455, func=0x2660a8 "ast_taskprocessor_get", cond_name=0x265f18 "&p->poll_cond", cond=0x2f0b34, cond_attr=0x0) at lock.c:454 #3 0x001f4340 in ast_taskprocessor_get (name=0x245250 "core_event_dispatcher", create=TPS_REF_DEFAULT) at taskprocessor.c:455 #4 0x00109354 in ast_event_init () at event.c:1873 #5 0x000665c4 in main (argc=2, argv=0xffffd964) at asterisk.c:4008 (gdb) cont Continuing. Program terminated with signal SIGBUS, Bus error. The program no longer exists. {noformat} By: Walter Doekes (wdoekes) 2013-04-24 02:23:13.472-0500 That's unexpected. I expected cond to land on an 8 byte boundary. Which it didn't (0x2f0b34). Google hints that the cause is misalignment of the &cond variable. I thought that placing it first would cause the alloc to happen on an 8 byte boundary. And I'd find it odd that the mem alloc of the struct would not land on said boundary. That leaves me with.. gcc reordering the fields? Please try this next patch. By: Walter Doekes (wdoekes) 2013-04-24 03:26:12.560-0500 And please check the address of p at the break: {noformat} up up up print p {noformat} If 'p' is misaligned, things will never work. By: Walter Doekes (wdoekes) 2013-04-24 03:30:50.202-0500 In case 'p' is misaligned, this might do the trick instead. By: Jeremy Kister (jkister) 2013-04-24 09:07:05.442-0500 it looks like you fixed it. applying patch 1 alone did not fix it applying patch 1+2 did not fix it. I have all three patches applied and asterisk 11.4.0-rc1 seems happy (admittedly, only up for a few minutes). By: Walter Doekes (wdoekes) 2013-04-24 09:34:34.835-0500 You needed *both* {{sparc_alignment_issues_2.diff}} and {{sparc_alignment_issues_3.diff}} ? Please check carefully which ones worked. * 2 alone should trump 1. So 1+2 should not be necessary. * 3 alone might work. If 2 is needed, then it's not funny: lots of code fixing/auditing ahead. * If 1 is really needed, that would be.. well.. unexpected. If you got the results back, then I'd welcome the next backtrace for the lingering issues. By: Jeremy Kister (jkister) 2013-04-24 10:43:45.357-0500 you're right; {{sparc_alignment_issues_3.diff}} alone makes asterisk start. when i start asterisk with /etc/init.d/asterisk, Asterisk starts but i see "Bus Error", but no core file. When I start asterisk with gdb /usr/sbin/asterisk & run -c , I don't get any Bus Error. What can I do to get /etc/init.d/asterisk to save core ? also: {noformat} pbx1.offphl> asterisk -r Asterisk 11.4.0-rc1, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Connected to Asterisk 11.4.0-rc1 currently running on pbx1 (pid = 28266) pbx1*CLI> sip show channels No such command 'sip show channels' (type 'core show help sip' for other possible commands) pbx1*CLI> core set debug 10 Core debug was 0 and is now 10 pbx1*CLI> core set verbose 10 Set remote console verbosity to 10 pbx1*CLI> module load chan_sip.so pbx1*CLI> sip show channels pbx1*CLI> pbx1*CLI> sip show peers pbx1*CLI> pbx1*CLI> core show warranty pbx1*CLI> core stop now pbx1*CLI> {noformat} By: Walter Doekes (wdoekes) 2013-04-24 11:12:30.429-0500 Isn't that rasterisk that dies? If asterisk still starts, the Bus Error probably comes from the "am I already running" check.. See the dumpcore= option in /etc/asterisk/asterisk.conf.. and use {{sh -x /etc/init.d/asterisk start}} to trace the init script. By: Jeremy Kister (jkister) 2013-04-24 11:23:29.809-0500 i feel bad making you jump through hoops to help me; just FYI i'm completely willing to give you ssh/root if you'd like. {noformat} pbx1.offphl> sh -x /etc/init.d/asterisk start + PATH=/usr/local/script:/usr/local/sbin:/usr/sbin:/usr/bin:/opt/SUNWspro/bin:/home/users/jkister/arch/Linux-2-sparc64/bin:/home//sys/scripts:/home//sys/bin:/opt/csw/bin:/opt/csw/gnu:/usr/sfw/bin:/usr/local/bin:/usr/local/script:/home/users/jkister/scripts:/usr/ccs/bin:/usr/dt/bin:/opt/gnu/bin:/usr/bin:/bin:/usr/ucb:/usr/lib:/usr/local/ezmlm/bin:/var/qmail/bin:/home/users/jkister/arch/Linux-2-sparc64/sbin:/home//sys/sbin:/opt/csw/sbin:/usr/sfw/sbin:/usr/local/sbin:/usr/sbin:/sbin:/usr/platform/unknown/sbin:. + export PATH + echo starting asterisk. starting asterisk. + nice -n -19 safe_asterisk Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?) Bus error (core dumped) pbx1> file core core: ELF 32-bit MSB core file, SPARC, version 1 (SYSV), SVR4-style, from 'rasterisk asterisk -nrx core show version' pbx1> gdb asterisk core GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. Reading symbols from /usr/lib/libasteriskssl.so.1...done. Loaded symbols for /usr/lib/libasteriskssl.so.1 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /usr/lib/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsqlite3.so.0 Reading symbols from /usr/lib/v9/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/v9/libssl.so.0.9.8 Reading symbols from /usr/lib/v9/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/v9/libcrypto.so.0.9.8 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libncurses.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libncurses.so.5 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Core was generated by `rasterisk asterisk -nrx core show version'. Program terminated with signal 10, Bus error. #0 0xf7df4e78 in ?? () from /lib/libc.so.6 (gdb) bt #0 0xf7df4e78 in ?? () from /lib/libc.so.6 #1 0xf7ef1008 in ?? () from /lib/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) {noformat} By: Walter Doekes (wdoekes) 2013-04-24 14:46:54.340-0500 Not jumping through hoops yet ;) I might take you up on that root access.. later A couple of things stand out: * It's indeed the version-check that crashes ('rasterisk asterisk -nrx core show version') * The backtrace is really poor this time around * You're generating 32 bit binaries on a 64 bit system ('ELF 32-bit MSB core file' and 'sparc64') I have no idea how/why you're making 32 bit binaries on a 64 bit system, but that looks like it's causing all sorts of havoc. The compiler aligns void-pointers on 4 byte boundaries, but mutex-related items are expected on 8 byte boundaries (it seems). Summarizing: * Asterisk starts with just the 3rd patch. * rasterisk core show version fails. Can you set some additional breakpoints when starting rasterisk? {noformat} gdb `which asterisk` break pthread_cond_init break pthread_attr_init break pthread_condattr_init break pthread_mutex_init break pthread_mutexattr_init break pthread_rwlock_init break pthread_rwlockattr_init run -nrx 'core show version' .. and then 'bt' and 'cont' until you find where the stack gets smashed {noformat} Alternately, we could alter the ast_mutex* macros to add logging of the pointer address. Depending on how quickly you can find another culprit using the method above. By: Jeremy Kister (jkister) 2013-04-24 15:13:18.376-0500 fyi, Asterisk 1.8.19.0 builds 32 bit on debian/sparc64 (and is working correctly) By: Rusty Newton (rnewton) 2013-04-24 18:04:26.611-0500 Jeremy in the guidelines https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines for various reasons we ask that debug and backtraces be attached as separate files and not included inline. Please remove your back traces and attach them when you can. In the comments you can link to the attachment [like this.|https://issues.asterisk.org/jira/secure/attachment/47054/modules.conf]. Thanks in advance! By: Walter Doekes (wdoekes) 2013-04-25 01:52:58.516-0500 @rnewton: these traces are short enough to fit inline. They're more readable this way than having to open separate attachments. @jkister: at breakpoint 5 things did not break yet.. you would have to {{cont}}-inue a bit to see what the last pthread_* function is that is called just before it dies. As for the fact that 1.8.21 works fine. My first theory would be that BUILD_NATIVE is disabled (or some other optimizing flag) on the 1.8 build. {noformat} touch main/astobj2.c make NOISY_BUILD=yes {noformat} What's the difference between the gcc flags for 1.8 vs. 11? By: Jeremy Kister (jkister) 2013-04-25 13:03:10.652-0500 for Asterisk 1.8.21.0: {noformat} pbx1.offphl> suex touch main/astobj2.c pbx1.offphl> suex make NOISY_BUILD=yes [...] make[1]: Entering directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main' gcc -o astobj2.o -c astobj2.c -MD -MT astobj2.o -MF .astobj2.o.d -MP -pthread -I/usr/local/src/asterisk/asterisk-1.8.21.0/include -pipe -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3 -O0 -DAST_MODULE=\"core\" cd editline && test -f config.h || CFLAGS="-pthread -I../..//include -pipe -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3 -O0 " LDFLAGS=" " ./configure --build=sparc64-unknown-linux-gnu --host=sparc64-unknown-linux-gnu --with-ncurses= --with-curses= --with-termcap= --with-tinfo= make -C editline libedit.a make[2]: Entering directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main/editline' make[2]: `libedit.a' is up to date. make[2]: Leaving directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main/editline' _ASTCFLAGS="-I/usr/local/src/asterisk/asterisk-1.8.21.0/include -pipe -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3 -O0 -Wno-strict-aliasing" ASTCFLAGS="" make -C db1-ast libdb1.a make[2]: Entering directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main/db1-ast' make[2]: `libdb1.a' is up to date. make[2]: Leaving directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main/db1-ast' gcc -o asterisk -Wl,--export-dynamic -Wl,--version-script,asterisk.exports -Wl,--dynamic-list,asterisk.dynamics abstract_jb.o acl.o adsi.o alaw.o aoc.o app.o ast_expr2.o ast_expr2f.o asterisk.o astfd.o astmm.o astobj2.o audiohook.o autochan.o autoservice.o bridging.o callerid.o ccss.o cdr.o cel.o channel.o chanvars.o cli.o config.o data.o datastore.o db.o devicestate.o dial.o dns.o dnsmgr.o dsp.o enum.o event.o features.o file.o fixedjitterbuf.o frame.o framehook.o fskmodem.o global_datastores.o hashtab.o heap.o http.o image.o indications.o io.o jitterbuf.o loader.o lock.o logger.o manager.o md5.o netsock.o netsock2.o pbx.o plc.o poll.o privacy.o rtp_engine.o say.o sched.o security_events.o sha1.o slinfactory.o srv.o ssl.o stdtime/localtime.o strcompat.o strings.o stun.o syslog.o taskprocessor.o tcptls.o tdd.o term.o test.o threadstorage.o timing.o translate.o udptl.o ulaw.o utils.o version.o xml.o xmldoc.o editline/libedit.a db1-ast/libdb1.a buildinfo.o -lssl -lcrypto -lc -ldl -lpthread -ltermcap -lm -lresolv make[1]: Leaving directory `/usr/local/src/asterisk/asterisk-1.8.21.0/main' +--------- Asterisk Build Complete ---------+ + Asterisk has successfully been built, and + + can be installed by running: + + + + make install + +-------------------------------------------+ {noformat} for Asterisk 11.4.0-rc1: {noformat} pbx1.offphl> suex touch main/astobj2.c pbx1.offphl> suex make NOISY_BUILD=yes CC="cc" CXX="" LD="" AR="" RANLIB="" CFLAGS="" LDFLAGS="" make -C menuselect CONFIGURE_SILENT="--silent" makeopts make[1]: Entering directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/menuselect' make[1]: `makeopts' is up to date. make[1]: Leaving directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/menuselect' CC="cc" CXX="" LD="" AR="" RANLIB="" CFLAGS="" LDFLAGS="" make -C menuselect CONFIGURE_SILENT="--silent" menuselect [...] make[1]: Entering directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/main' gcc -o astobj2.o -c astobj2.c -MD -MT astobj2.o -MF .astobj2.o.d -MP -pthread -I/usr/local/src/asterisk/asterisk-11.4.0-rc1/include -pipe -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3 -O0 -DAST_MODULE=\"core\" cd editline && test -f config.h || CFLAGS="-pthread -I../..//include -pipe -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -g3 -O0 " LDFLAGS=" " ./configure --build=sparc64-unknown-linux-gnu --host=sparc64-unknown-linux-gnu --with-ncurses= --with-curses= --with-termcap= --with-tinfo= make -C editline libedit.a make[2]: Entering directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/main/editline' make[2]: `libedit.a' is up to date. make[2]: Leaving directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/main/editline' gcc -o asterisk -Wl,--export-dynamic -Wl,--version-script,asterisk.exports -Wl,--dynamic-list,asterisk.dynamics abstract_jb.o acl.o adsi.o alaw.o aoc.o app.o ast_expr2.o ast_expr2f.o asterisk.o astfd.o astmm.o astobj2.o audiohook.o autochan.o autoservice.o bridging.o callerid.o ccss.o cdr.o cel.o channel.o channel_internal_api.o chanvars.o cli.o config.o config_options.o data.o datastore.o db.o devicestate.o dial.o dns.o dnsmgr.o dsp.o enum.o event.o features.o file.o fixedjitterbuf.o format.o format_cap.o format_pref.o frame.o framehook.o fskmodem.o global_datastores.o hashtab.o heap.o http.o image.o indications.o io.o jitterbuf.o loader.o lock.o logger.o manager.o md5.o message.o named_acl.o netsock.o netsock2.o pbx.o plc.o poll.o presencestate.o privacy.o rtp_engine.o say.o sched.o security_events.o sha1.o sip_api.o slinfactory.o srv.o stdtime/localtime.o strcompat.o strings.o stun.o syslog.o taskprocessor.o tcptls.o tdd.o term.o test.o threadstorage.o timing.o translate.o udptl.o ulaw.o utils.o version.o xml.o xmldoc.o -L. -lasteriskssl editline/libedit.a buildinfo.o -lc -lsqlite3 -lssl -lcrypto -ldl -lpthread -ltermcap -lm -lresolv make[1]: Leaving directory `/usr/local/src/asterisk/asterisk-11.4.0-rc1/main' +--------- Asterisk Build Complete ---------+ + Asterisk has successfully been built, and + + can be installed by running: + + + + make install + +-------------------------------------------+ {noformat} By: Jeremy Kister (jkister) 2013-04-25 13:11:05.980-0500 oh, sorry about stopping early. i didnt realize what i was looking for. {noformat} pbx1.offphl> gdb `which asterisk` GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "sparc-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/asterisk...done. (gdb) break pthread_cond_init Breakpoint 1 at 0x260b08 (gdb) break pthread_attr_init Breakpoint 2 at 0x260bbc (gdb) break pthread_condattr_init Function "pthread_condattr_init" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 3 (pthread_condattr_init) pending. (gdb) break pthread_mutex_init Breakpoint 4 at 0x260ec8 (gdb) break pthread_mutexattr_init Breakpoint 5 at 0x2613fc (gdb) break pthread_rwlock_init Breakpoint 6 at 0x2606dc (gdb) break pthread_rwlockattr_init Breakpoint 7 at 0x2606e8 (gdb) run -nrx 'core show version' Starting program: /usr/sbin/asterisk -nrx 'core show version' [Thread debugging using libthread_db enabled] Breakpoint 5, 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 #1 0x0014ac10 in ast_reentrancy_init (plt=0x264a08) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:473 #2 0x0014e03c in __ast_rwlock_wrlock (filename=0x21d0d8 "asterisk.c", line=349, func=0x221850 "ast_register_file_version", t=0x2649e8, name=0x21d108 "&(&file_versions)->lock") at lock.c:998 #3 0x00055f84 in ast_register_file_version (file=0xf7fb54f8 "libasteriskssl.c", version=0xf7fb5510 "$Revision: 373079 $") at asterisk.c:349 #4 0xf7fb4c28 in __register_file_version () at libasteriskssl.c:32 #5 0xf7fb5494 in __do_global_ctors_aux () from /usr/lib/libasteriskssl.so.1 #6 0xf7fb4a30 in _init () from /usr/lib/libasteriskssl.so.1 #7 0xf7fd8c4c in ?? () from /lib/ld-linux.so.2 #8 0xf7fd8c4c in ?? () from /lib/ld-linux.so.2 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) cont Continuing. Breakpoint 4, 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 (gdb) bt #0 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 #1 0x0014ac40 in ast_reentrancy_init (plt=0x264a08) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:475 #2 0x0014e03c in __ast_rwlock_wrlock (filename=0x21d0d8 "asterisk.c", line=349, func=0x221850 "ast_register_file_version", t=0x2649e8, name=0x21d108 "&(&file_versions)->lock") at lock.c:998 #3 0x00055f84 in ast_register_file_version (file=0xf7fb54f8 "libasteriskssl.c", version=0xf7fb5510 "$Revision: 373079 $") at asterisk.c:349 #4 0xf7fb4c28 in __register_file_version () at libasteriskssl.c:32 #5 0xf7fb5494 in __do_global_ctors_aux () from /usr/lib/libasteriskssl.so.1 #6 0xf7fb4a30 in _init () from /usr/lib/libasteriskssl.so.1 #7 0xf7fd8c4c in ?? () from /lib/ld-linux.so.2 #8 0xf7fd8c4c in ?? () from /lib/ld-linux.so.2 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) cont Continuing. Breakpoint 5, 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 #1 0x0014ac10 in ast_reentrancy_init (plt=0x266380) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:473 #2 0x0014b454 in __ast_pthread_mutex_lock (filename=0x232610 "config.c", lineno=1568, func=0x233600 "config_text_file_load", mutex_name=0x2326d8 "&(&cfmtime_head)->lock", t=0x266368) at lock.c:188 #3 0x000d4a68 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1568 #4 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #5 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #6 0x00060428 in ast_readconfig () at asterisk.c:3132 #7 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 4, 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 (gdb) bt #0 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 #1 0x0014ac40 in ast_reentrancy_init (plt=0x266380) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:475 #2 0x0014b454 in __ast_pthread_mutex_lock (filename=0x232610 "config.c", lineno=1568, func=0x233600 "config_text_file_load", mutex_name=0x2326d8 "&(&cfmtime_head)->lock", t=0x266368) at lock.c:188 #3 0x000d4a68 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1568 #4 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #5 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #6 0x00060428 in ast_readconfig () at asterisk.c:3132 #7 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 5, 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 #1 0x0014ac10 in ast_reentrancy_init (plt=0x26ab7c) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:473 #2 0x0014b454 in __ast_pthread_mutex_lock (filename=0x257640 "stdtime/localtime.c", lineno=1466, func=0x2578e0 "ast_tzset", mutex_name=0x257680 "&(&zonelist)->lock", t=0x26ab64) at lock.c:188 #3 0x001e26d8 in ast_tzset (zone=0x257838 "/etc/localtime") at stdtime/localtime.c:1466 #4 0x001e2f14 in ast_localtime (timep=0xffff96e4, tmp=0xffff96ec, zone=0x0) at stdtime/localtime.c:1591 #5 0x001545f0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1505 #6 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #7 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #8 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #9 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #10 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #11 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #12 0x00060428 in ast_readconfig () at asterisk.c:3132 #13 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 4, 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 (gdb) bt #0 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 #1 0x0014ac40 in ast_reentrancy_init (plt=0x26ab7c) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:475 #2 0x0014b454 in __ast_pthread_mutex_lock (filename=0x257640 "stdtime/localtime.c", lineno=1466, func=0x2578e0 "ast_tzset", mutex_name=0x257680 "&(&zonelist)->lock", t=0x26ab64) at lock.c:188 #3 0x001e26d8 in ast_tzset (zone=0x257838 "/etc/localtime") at stdtime/localtime.c:1466 #4 0x001e2f14 in ast_localtime (timep=0xffff96e4, tmp=0xffff96ec, zone=0x0) at stdtime/localtime.c:1591 #5 0x001545f0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1505 #6 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #7 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #8 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #9 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #10 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #11 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #12 0x00060428 in ast_readconfig () at asterisk.c:3132 #13 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 5, 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 #1 0x0014ac10 in ast_reentrancy_init (plt=0x268fb8) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:473 #2 0x0014dc48 in __ast_rwlock_rdlock (filename=0x244378 "logger.c", line=1032, func=0x245508 "logger_print_normal", t=0x268f98, name=0x244e38 "&(&verbosers)->lock") at lock.c:879 #3 0x00152abc in logger_print_normal (logmsg=0x2a4700) at logger.c:1032 #4 0x00154dd0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1529 #5 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #6 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #7 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #8 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #9 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #10 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #11 0x00060428 in ast_readconfig () at asterisk.c:3132 #12 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 4, 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 (gdb) bt #0 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 #1 0x0014ac40 in ast_reentrancy_init (plt=0x268fb8) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:475 #2 0x0014dc48 in __ast_rwlock_rdlock (filename=0x244378 "logger.c", line=1032, func=0x245508 "logger_print_normal", t=0x268f98, name=0x244e38 "&(&verbosers)->lock") at lock.c:879 #3 0x00152abc in logger_print_normal (logmsg=0x2a4700) at logger.c:1032 #4 0x00154dd0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1529 #5 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #6 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #7 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #8 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #9 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #10 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #11 0x00060428 in ast_readconfig () at asterisk.c:3132 #12 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 5, 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 (gdb) bt #0 0xf7b8eb00 in pthread_mutexattr_init () from /lib/libpthread.so.0 #1 0x0014ac10 in ast_reentrancy_init (plt=0x268ebc) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:473 #2 0x0014dc48 in __ast_rwlock_rdlock (filename=0x244378 "logger.c", line=1039, func=0x245508 "logger_print_normal", t=0x268e9c, name=0x2445a0 "&(&logchannels)->lock") at lock.c:879 #3 0x00152cf4 in logger_print_normal (logmsg=0x2a4700) at logger.c:1039 #4 0x00154dd0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1529 #5 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #6 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #7 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #8 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #9 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #10 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #11 0x00060428 in ast_readconfig () at asterisk.c:3132 #12 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Breakpoint 4, 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 (gdb) bt #0 0xf7f2eda4 in pthread_mutex_init () from /lib/libc.so.6 #1 0x0014ac40 in ast_reentrancy_init (plt=0x268ebc) at /usr/local/src/asterisk/asterisk-11.4.0-rc1/include/asterisk/lock.h:475 #2 0x0014dc48 in __ast_rwlock_rdlock (filename=0x244378 "logger.c", line=1039, func=0x245508 "logger_print_normal", t=0x268e9c, name=0x2445a0 "&(&logchannels)->lock") at lock.c:879 #3 0x00152cf4 in logger_print_normal (logmsg=0x2a4700) at logger.c:1039 #4 0x00154dd0 in ast_log_full (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s", ap=0xffff9834) at logger.c:1529 #5 0x00154ef0 in ast_log_callid (level=5, file=0x232610 "config.c", line=1645, function=0x233600 "config_text_file_load", callid=0x0, fmt=0x244830 "%s") at logger.c:1554 #6 0x001557b0 in __ast_verbose_ap (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, callid=0x0, fmt=0xffff9840 "\375 == Parsing '%s': Found\n", ap=0xffff99b0) at logger.c:1822 #7 0x0015582c in __ast_verbose (file=0x232610 "config.c", line=1645, func=0x233600 "config_text_file_load", level=2, fmt=0x232b30 "Parsing '%s': Found\n") at logger.c:1833 #8 0x000d52d0 in config_text_file_load (database=0xffffc0f8 "", table=0xffffbff8 "", filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:1645 #9 0x000d8f2c in ast_config_internal_load (filename=0x2203c0 "/etc/asterisk/asterisk.conf", cfg=0x2a3cb8, flags=..., suggested_include_file=0x232720 "", who_asked=0x21fc30 "") at config.c:2496 #10 0x000d9030 in ast_config_load2 (filename=0x2203c0 "/etc/asterisk/asterisk.conf", who_asked=0x21fc30 "", flags=...) at config.c:2517 #11 0x00060428 in ast_readconfig () at asterisk.c:3132 #12 0x00063c30 in main (argc=3, argv=0xffffd944) at asterisk.c:3662 (gdb) cont Continuing. Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?) Program received signal SIGBUS, Bus error. 0xf7eb0e78 in ?? () from /lib/libc.so.6 (gdb) {noformat} By: Walter Doekes (wdoekes) 2013-04-25 15:03:25.923-0500 Ok. The bt/cont loop suggests that the ast_mutext_t in linkedlist.h should also get an aligned attribute. However.. that would mean that all of the following would need adapting too. All ast_mutex_t/ast_rwlock_t in any struct. {noformat} $ find . -name '*.h' -or -name '*.c' | xargs grep '^[[:blank:]]*struct[^;]*$' -A30 | egrep 'ast_(mutex|rwlock)_t' | sort ... ./include/asterisk/linkedlists.h- ast_mutex_t lock; \ ./include/asterisk/linkedlists.h- ast_rwlock_t lock; \ ... ./main/astobj2.c- ast_mutex_t lock; ./main/astobj2.c- ast_rwlock_t lock; ./main/data.c- ast_rwlock_t lock; ./main/devicestate.c- ast_mutex_t lock; ... $ find . -name '*.h' -or -name '*.c' | xargs grep '^[[:blank:]]*struct[^;]*$' -A30 | egrep 'ast_(mutex|rwlock)_t' | wc -l 112 $ find . -name '*.h' -or -name '*.c' | xargs grep '^[[:blank:]]*struct[^;]*$' -A30 | egrep 'ast_(mutex|rwlock)_t' | sort | sed -e 's/^..//;s/\/.*//' | uniq -c 5 addons 11 apps 1 bridges 2 cdr 1 cel 25 channels 1 funcs 27 include 20 main 7 res 12 utils {noformat} That's no fun to fix. (And then I even skipped the ast_cond_t's.) That can't be right. I must be looking at this from the wrong angle. What's different between 1.8 and 11? That change should be significantly less intrusive. Unfortunately I'm out of ideas there. The two noisy builds you've shown both show identical CC and LD flags. By: Jeremy Kister (jkister) 2013-04-26 13:50:39.622-0500 FYI, it appears vanilla Asterisk 10.12.2 works correctly, save the similar Bus Error while using rastersik: {noformat} pbx1.offphl> ps -ef | grep ast root 25091 25063 0 14:45 pts/1 00:00:00 grep ast pbx1.offphl> /etc/init.d/asterisk start starting asterisk. Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?) Bus error (core dumped) pbx1.offphl> ps -ef | grep ast root 25106 1 0 14:45 pts/1 00:00:00 /bin/sh /usr/sbin/safe_asterisk root 25108 25106 77 14:45 pts/1 00:00:00 /usr/sbin/asterisk -f -vvvg -c root 25126 25063 0 14:45 pts/1 00:00:00 grep ast pbx1.offphl> asterisk -r Asterisk 10.12.2, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Connected to Asterisk 10.12.2 currently running on pbx1 (pid = 25108) Verbosity is at least 3 pbx1*CLI> core show version Asterisk 10.12.2 built by root @ pbx1.offphl on a sparc64 running Linux on 2013-04-26 17:55:09 UTC pbx1*CLI> quit Executing last minute cleanups pbx1.offphl> asterisk -rx 'core show version' Asterisk 10.12.2 built by root @ pbx1.offphl on a sparc64 running Linux on 2013-04-26 17:55:09 UTC pbx1.offphl> {noformat} By: Jeremy Kister (jkister) 2013-04-26 14:12:10.932-0500 ohhhh, at closer look, it appears 1.8.20.0's /etc/init.d/asterisk start does the same thing - so that is a separate issue all together. see ASTERISK-21763 for the rasterisk bus error. By: Walter Doekes (wdoekes) 2013-05-10 03:56:03.784-0500 Ok. So we only need a cleaned up version of {{sparc_alignment_issues_3.diff}} for asterisk 11 and up to take care of this ticket? By: Jeremy Kister (jkister) 2013-05-15 09:27:32.843-0500 it depends how you want to deal with more issues. this ticket was originally created because Asterisk would not even start. with your [^sparc_alignment_issues_3.diff], Asterisk 11 does start on sparc, but several modules (including chan_sip) don't load. So, It depends if you'd want the modules as separate a issue or part of this one. By: Walter Doekes (wdoekes) 2013-05-15 09:33:58.486-0500 Ok. I think I'm going to need that shell access if I'm to debug chan_sip not loading. If you're still ok with that, you could pass me credentials over IRC or e-mail. Can't promise when I have time to take a look though. By: Walter Doekes (wdoekes) 2013-06-19 09:35:50.336-0500 For the record: {noformat} root@t1000:/usr/local/src/sparcbus# cat voidsize.c int main() { return sizeof(void*); } {noformat} {noformat} root@t1000:/usr/local/src/sparcbus# ./voidsize; echo $? 4 root@t1000:/usr/local/src/sparcbus# uname -a Linux t1000 3.2.0-4-sparc64-smp #1 SMP Debian 3.2.41-2+deb7u2 sparc64 GNU/Linux {noformat} Notice how the void* is sized 4 and aligned on 4 byte boundaries, even though this is a 64 bit system (requiring 8-byte aligned "cond_t"'s). That's what makes {{sparc_alignment_issues_3.diff}} necessary. {noformat}__alignof__(pthread_cond_t) == 8{noformat} By: Walter Doekes (wdoekes) 2013-06-19 09:41:42.957-0500 Jeremy: on your t1000, you run with 2 patches now: the {{sparc_alignment_issues_3.diff}} and the {{asterisk.exports}} patch. (And nothing more?) Can you check/test whether there is any breakage left? By: Jeremy Kister (jkister) 2013-06-19 22:26:43.272-0500 fyi, asterisk 1.8 seems totally happy with {{asterisk.exports}} patch asterisk 11 seems still upset with {{sparc_alignment_issues_3.diff}} and {{asterisk.exports}} patch: {noformat} root@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1# asterisk -r Asterisk 11.5.0-rc1, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Connected to Asterisk 11.5.0-rc1 currently running on t1000 (pid = 64575) t1000*CLI> module show like chan_sip Module Description Use Count chan_sip.so Session Initiation Protocol (SIP) 0 1 modules loaded t1000*CLI> sip show peers No such command 'sip show peers' (type 'core show help sip' for other possible commands) t1000*CLI> core show warranty t1000*CLI> help t1000*CLI> foo t1000*CLI> quit Asterisk cleanly ending (0). Executing last minute cleanups root@t1000:/usr/local/src/asterisk/asterisk-11.5.0-rc1# {noformat} also note this odd behavior: {noformat} t1000*CLI> module show like sip Module Description Use Count app_adsiprog.so Asterisk ADSI Programming Application 0 chan_sip.so Session Initiation Protocol (SIP) 0 2 modules loaded t1000*CLI> module unload chan_sip.so t1000*CLI> module show like sip t1000*CLI> quit jkister@t1000:~$ suex asterisk -r Asterisk 11.5.0-rc1, Copyright (C) 1999 - 2012 Digium, Inc. and others. Created by Mark Spencer <markster@digium.com> Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. This is free software, with components licensed under the GNU General Public License version 2 and other licenses; you are welcome to redistribute it under certain conditions. Type 'core show license' for details. ========================================================================= Connected to Asterisk 11.5.0-rc1 currently running on t1000 (pid = 64575) t1000*CLI> module show like sip Module Description Use Count app_adsiprog.so Asterisk ADSI Programming Application 0 chan_sip.so Session Initiation Protocol (SIP) 0 2 modules loaded t1000*CLI> module unload chan_sip.so t1000*CLI> module load chan_sip.so t1000*CLI> module show like sip t1000*CLI> quit {noformat} By: Walter Doekes (wdoekes) 2013-08-22 03:32:17.579-0500 Ok. So 1.8 is fixed and committed. And 11 needs: (a) {{sparc_alignment_issues_3.diff}} But is then still broken? Does (a) do anything at all? Is it worse without it? (b) Did you check whether adding {{sparc_alignment_issues.diff}} and/or {{sparc_alignment_issues_3.diff}} and/or the the put_aligned32 patch from the other ticket does anything to improve/worsen the situation? Cheers, Walter By: Jeremy Kister (jkister) 2013-08-27 00:20:48.893-0500 [^sparc_alignment_issues_3.diff] as well as _IO_stdin_used gets asterisk 11 for sparc much happier -- asterisk starts, all modules i've seen problems with in the past are now behaving normally. however, there remain two obscure issues: 1. when trying to stop asterisk with "asterisk -rx 'core stop now'", the command sits for a bit and eventually crashes. see [^core_stop_now_backtrace.txt], but see issue with backtrace in #2. 2. when compiling with DEBUG_THREADS [seemingly] random modules do not load. even 'core show warranty' prints "no such command". [^sparc_alignment_issues.diff] does not appear to have an effect on the above issues. By: Richard Mudgett (rmudgett) 2013-08-27 10:14:27.459-0500 {quote} 2. when compiling with DEBUG_THREADS [seemingly] random modules do not load. even 'core show warranty' prints "no such command". {quote} I just recently (as in last week) fixed a memory corruption issue with DEBUG_THREADS in v1.8 and v11. By: Jeremy Kister (jkister) 2013-08-27 16:02:20.861-0500 @richard can you post a reviewboard link? I'm having trouble finding it. Thanks! By: Richard Mudgett (rmudgett) 2013-08-27 20:40:03.840-0500 The link is: https://reviewboard.asterisk.org/r/2778/ By: Walter Doekes (wdoekes) 2013-08-29 01:56:28.673-0500 Jeremy: if you pull the latest 11.x from SVN, you get both _IO_stdin_used and rb2778 (commited in r397528). You'd only need to re-apply sparc..3.diff. (*) Do a make distclean and re-run configure, just to be on the safe side. By: Jeremy Kister (jkister) 2013-08-30 00:51:01.479-0500 i just pulled SVN-branch-11-r397948M then applied [^sparc_alignment_issues_3.diff]. both problems still exist: {quote} 1. when trying to stop asterisk with "asterisk -rx 'core stop now'", the command sits for a bit and eventually crashes. see [^core_stop_now_backtrace.txt], but see issue with backtrace in #2. 2. when compiling with DEBUG_THREADS [seemingly] random modules do not load. even 'core show warranty' prints "no such command". {quote} By: Walter Doekes (wdoekes) 2013-08-30 03:12:17.990-0500 Ok. Thanks for checking that. When I get the time, I'll go poke around some more on your machine. By: Walter Doekes (wdoekes) 2013-09-02 03:31:38.941-0500 I did some poking last friday. And the cause of the 'core stop now' crash was a problem with allocation at the start. First when freeing that memory it crashes. The problem -- which I cannot explain at all -- is that malloc(3) starts returning 4-byte aligned addresses after a while. And that obviously causes trouble. I added a couple of assertions (*) that alloc'ed memory should be 8-byte aligned, and it always trips on the same allocation during initialization. Changing size of the allocation didn't help. (*) With abort() so you can debug the surroundings with gdb. For some reason running asterisk under GDB refuses to work (something with libthread-debug stuff). By: Walter Doekes (wdoekes) 2013-11-22 07:48:12.504-0600 A quick google search turned up this today: http://gcc.gnu.org/onlinedocs/gcc/Interoperation.html {quote} On Solaris, the malloc function in the libmalloc.a library may allocate memory that is only 4 byte aligned. Since GCC on the SPARC assumes that doubles are 8 byte aligned, this may result in a fatal signal if doubles are stored in memory allocated by the libmalloc.a library. The solution is to not use the libmalloc.a library. Use instead malloc and related functions from libc.a; they do not have this problem. {quote} That might be something to go on when I have time to play around with it. By: Walter Doekes (wdoekes) 2016-02-12 03:18:37.194-0600 Is this still an issue? By: Jeremy Kister (jkister) 2016-02-12 07:57:49.741-0600 I dont have sparc any more, cannot confirm. By: Walter Doekes (wdoekes) 2016-02-12 08:21:06.618-0600 Thanks. I'll just close it then :) | ||||||